-
Notifications
You must be signed in to change notification settings - Fork 495
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Migrate distributors hash ring to BasicLifecycler and add auto-forget feature #2138
Labels
Comments
56quarters
added a commit
that referenced
this issue
Jun 21, 2022
Use the BasicLifecycler in distributors for managing their lifecycle so that we can take advantage of the "auto-forget" delegates feature. This prevents the ring from filling up with "unhealthy" distributors that are never removed. This wasn't a bug but it was confusing for users and operators. Fixes #2138 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
2 tasks
56quarters
added a commit
that referenced
this issue
Jun 21, 2022
Use the BasicLifecycler in distributors for managing their lifecycle so that we can take advantage of the "auto-forget" delegates feature. This prevents the ring from filling up with "unhealthy" distributors that are never removed. This wasn't a bug but it was confusing for users and operators. Fixes #2138 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
56quarters
added a commit
that referenced
this issue
Jun 22, 2022
Use the BasicLifecycler in distributors for managing their lifecycle so that we can take advantage of the "auto-forget" delegates feature. This prevents the ring from filling up with "unhealthy" distributors that are never removed. This wasn't a bug but it was confusing for users and operators. Fixes #2138 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
56quarters
added a commit
that referenced
this issue
Jun 23, 2022
Use the BasicLifecycler in distributors for managing their lifecycle so that we can take advantage of the "auto-forget" delegates feature. This prevents the ring from filling up with "unhealthy" distributors that are never removed. This wasn't a bug but it was confusing for users and operators. Fixes #2138 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
56quarters
added a commit
that referenced
this issue
Jun 24, 2022
Use the BasicLifecycler in distributors for managing their lifecycle so that we can take advantage of the "auto-forget" delegates feature. This prevents the ring from filling up with "unhealthy" distributors that are never removed. This wasn't a bug but it was confusing for users and operators. Fixes #2138 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
56quarters
added a commit
that referenced
this issue
Jun 24, 2022
Use the BasicLifecycler in distributors for managing their lifecycle so that we can take advantage of the "auto-forget" delegates feature. This prevents the ring from filling up with "unhealthy" distributors that are never removed. This wasn't a bug but it was confusing for users and operators. Fixes #2138 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
masonmei
pushed a commit
to udmire/mimir
that referenced
this issue
Jul 11, 2022
Use the BasicLifecycler in distributors for managing their lifecycle so that we can take advantage of the "auto-forget" delegates feature. This prevents the ring from filling up with "unhealthy" distributors that are never removed. This wasn't a bug but it was confusing for users and operators. Fixes grafana#2138 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When a distributor crash and is restarted with a different instance ID, it will never leave the hash ring and the instance will be in the hash ring with "Unhealthy" state until an human operator will manually forget through the admin UI. The "Unhealthy" distributor in the ring does not cause any issue in the Mimir cluster, but it's misleading for users and we should fix it.
The distributors hash ring is still implemented using the old
ring.Lifecycler
:mimir/pkg/distributor/distributor.go
Line 220 in e11ac85
We should:
ring.BasicLifecycler
The text was updated successfully, but these errors were encountered: