-
Notifications
You must be signed in to change notification settings - Fork 477
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make the ring auto-forget distributors which have been in an unhealthy state for too long #571
Comments
@replay Can you provide more tips please? I would like to implement it, at the moment I don't know how to handle the ingestionRateStrategy properly, it looks like BasicLifecycler doesn't implement ReadLifecycler. |
Hi @hi-rustin, Thanks for looking into this! mimir/pkg/storegateway/gateway.go Line 146 in d2ae62f
But to use that feature the ring Distributor will have to use a |
I wonder if I can keep track of it via OnRingInstanceRegister and OnRingInstanceStopping? |
I'm not sure that's sufficient, as the health status can also change at other times than when an instance is registering or stopping. So maybe the same thing could be done in the |
Done in #2154. |
We should make the distributor ring auto forget the distributor instances which have been unhealthy for a defined amount of time because when a distributor gets hard killed then it has no chance to unregister itself from the ring. Over time this can lead to a situation where the ring is full of old unhealthy instances, growing the size of the ring and putting load on the memberlist client.
Since the distributors have no state there should be no risk associated with forgetting about the unhealthy ones.
Distributors use
ring.Lifecycler
while the code we already have requiresring.BasicLifecycler
. We should move distributors toring.BasicLifecycler
and use the auto-forget feature.The text was updated successfully, but these errors were encountered: