New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memberlist: Aggressive scale down can cause lost tombstones #4360
Labels
Comments
stevesg
added a commit
to stevesg/cortex
that referenced
this issue
Jul 12, 2021
This appears to be highlighting an issue - raised as cortexproject#4360. This change just stops the test flaking until it can be fixed. Signed-off-by: Steve Simpson <steve.simpson@grafana.com>
3 tasks
pracucci
pushed a commit
that referenced
this issue
Jul 13, 2021
This appears to be highlighting an issue - raised as #4360. This change just stops the test flaking until it can be fixed. Signed-off-by: Steve Simpson <steve.simpson@grafana.com>
This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions. |
alvinlin123
pushed a commit
to ac1214/cortex
that referenced
this issue
Jan 14, 2022
…xproject#4361) This appears to be highlighting an issue - raised as cortexproject#4360. This change just stops the test flaking until it can be fixed. Signed-off-by: Steve Simpson <steve.simpson@grafana.com> Signed-off-by: Alvin Lin <alvinlin@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
When scaling down extremely fast, a tombstone can still go missing. The
TestSingleBinaryWithMemberlistScaling
can reproduce this on occasion with the default values. e.g.memberlist-tombstone-with-debug.log
What appears to be happening is that the final messages from the instance being scaled down are being sent the expected number of times, but the intended recipients are also shutting down. This is not trivial to fix because we do not get any feedback from memberlist as to whether our messages were actually received. Possible solutions:
To Reproduce
Run the
TestSingleBinaryWithMemberlistScaling
a few times.Tweaking the scaling numbers in the test make it fail more often:
Expected behavior
The test doesn't fail.
Environment:
Additional Context
The text was updated successfully, but these errors were encountered: