-
Notifications
You must be signed in to change notification settings - Fork 791
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix ingesters with less tokens stuck in LEAVING #5061
Fix ingesters with less tokens stuck in LEAVING #5061
Conversation
Signed-off-by: Mariana Franco <marfram@amazon.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I don't have time to improve these unit tests now. I can create an issue if you want. Hopefully other contributors can help to improve the coverage. |
Signed-off-by: Mariana Franco <marfram@amazon.com>
Signed-off-by: Friedrich Gonzalez <friedrichg@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
@alanprot Could you take a look? |
Signed-off-by: Mariana Franco marfram@amazon.com
What this PR does:
It's possible to have ingesters in the ring with less tokens than configured because of conflict resolution:
cortex/pkg/ring/model.go
Lines 354 to 361 in f930c7a
In this case, the ingester will get stuck in LEAVING when they are evicted and rescheduled.
Looking at previous changes, seems that there was not a good reason to check for NumTokens: #1921 (comment)
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]