Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate distributors hash ring to BasicLifecycler and add auto-forget feature #2138

Closed
pracucci opened this issue Jun 20, 2022 · 0 comments · Fixed by #2154
Closed

Migrate distributors hash ring to BasicLifecycler and add auto-forget feature #2138

pracucci opened this issue Jun 20, 2022 · 0 comments · Fixed by #2154
Assignees
@56quarters
Labels
ease-of-use enhancement New feature or request

Comments

@pracucci
Copy link
Collaborator

When a distributor crash and is restarted with a different instance ID, it will never leave the hash ring and the instance will be in the hash ring with "Unhealthy" state until an human operator will manually forget through the admin UI. The "Unhealthy" distributor in the ring does not cause any issue in the Mimir cluster, but it's misleading for users and we should fix it.

The distributors hash ring is still implemented using the old ring.Lifecycler:

mimir/pkg/distributor/distributor.go

Line 220 in e11ac85

distributorsLifeCycler, err = ring.NewLifecycler(cfg.DistributorRing.ToLifecyclerConfig(), nil, "distributor", DistributorRingKey, true, log, prometheus.WrapRegistererWithPrefix("cortex_", reg))

We should:

  1. Migrate the distributor hash ring to ring.BasicLifecycler
  2. Enable the auto-forget feature
@pracucci pracucci added enhancement New feature or request ease-of-use labels Jun 20, 2022
@56quarters 56quarters self-assigned this Jun 20, 2022
56quarters added a commit that referenced this issue Jun 21, 2022
@56quarters
Use BasicLifecycler for distributors and auto-forget
5a6741d 
Use the BasicLifecycler in distributors for managing their lifecycle so
that we can take advantage of the "auto-forget" delegates feature. This
prevents the ring from filling up with "unhealthy" distributors that are
never removed. This wasn't a bug but it was confusing for users and
operators.

Fixes #2138

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
@56quarters 56quarters mentioned this issue Jun 21, 2022
Merged
2 tasks
56quarters added a commit that referenced this issue Jun 21, 2022
@56quarters
Use BasicLifecycler for distributors and auto-forget
2a5c10c 
Use the BasicLifecycler in distributors for managing their lifecycle so
that we can take advantage of the "auto-forget" delegates feature. This
prevents the ring from filling up with "unhealthy" distributors that are
never removed. This wasn't a bug but it was confusing for users and
operators.

Fixes #2138

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
56quarters added a commit that referenced this issue Jun 22, 2022
@56quarters
Use BasicLifecycler for distributors and auto-forget
a096530 
Use the BasicLifecycler in distributors for managing their lifecycle so
that we can take advantage of the "auto-forget" delegates feature. This
prevents the ring from filling up with "unhealthy" distributors that are
never removed. This wasn't a bug but it was confusing for users and
operators.

Fixes #2138

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
56quarters added a commit that referenced this issue Jun 23, 2022
@56quarters
Use BasicLifecycler for distributors and auto-forget
83c065d 
Use the BasicLifecycler in distributors for managing their lifecycle so
that we can take advantage of the "auto-forget" delegates feature. This
prevents the ring from filling up with "unhealthy" distributors that are
never removed. This wasn't a bug but it was confusing for users and
operators.

Fixes #2138

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
56quarters added a commit that referenced this issue Jun 24, 2022
@56quarters
Use BasicLifecycler for distributors and auto-forget
dd6367e 
Use the BasicLifecycler in distributors for managing their lifecycle so
that we can take advantage of the "auto-forget" delegates feature. This
prevents the ring from filling up with "unhealthy" distributors that are
never removed. This wasn't a bug but it was confusing for users and
operators.

Fixes #2138

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
56quarters added a commit that referenced this issue Jun 24, 2022
@56quarters
Use BasicLifecycler for distributors and auto-forget (#2154)
7e4accd 
Use the BasicLifecycler in distributors for managing their lifecycle so
that we can take advantage of the "auto-forget" delegates feature. This
prevents the ring from filling up with "unhealthy" distributors that are
never removed. This wasn't a bug but it was confusing for users and
operators.

Fixes #2138

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
masonmei pushed a commit to udmire/mimir that referenced this issue Jul 11, 2022
@56quarters
Use BasicLifecycler for distributors and auto-forget (grafana#2154)
4247214 
Use the BasicLifecycler in distributors for managing their lifecycle so
that we can take advantage of the "auto-forget" delegates feature. This
prevents the ring from filling up with "unhealthy" distributors that are
never removed. This wasn't a bug but it was confusing for users and
operators.

Fixes grafana#2138

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@56quarters 56quarters

Labels
ease-of-use enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use BasicLifecycler for distributors and auto-forget grafana/mimir
2 participants
@56quarters @pracucci