Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ring view accessed via ingester shows unhealthy ingesters, even if they are healthy #1917

Closed
pstibrany opened this issue May 24, 2022 · 0 comments · Fixed by grafana/dskit#172 or #1933
Assignees
Labels
bug Something isn't working component/hash-ring good first issue Good for newcomers

Comments

@pstibrany
Copy link
Member

Describe the bug

In #654 we have introduced ability to view ingester ring state via endpoint in ingesters. This endpoint however uses "heartbeat period" for "heartbeat timeout" when rendering the page, which is a bug. Hearbeat period is often much shorter (eg. it defaults to 5s for -ingester.ring.heartbeat-period flag) than heartbeat timeout (-ingester.ring.heartbeat-timeout defaults to 1min), so ingesters show up as "Unhealthy", even if they are not.

Link to the code: https://github.com/grafana/dskit/blob/25baa36b7a6fca2025c4e15d7e95d4810c85fa16/ring/http.go#L109

Notice how lifecycler passes HeartbeatPeriod to the ring handler, while "ring.go" used by distributor passes correct HeartbeatTimeout:

Screenshot 2022-05-24 at 12 12 03

To Reproduce

  1. Start Mimir (SHA or version)
  2. Access /ingester/ring endpoint on ingesters, notice unhealthy ingesters.
  3. Access /ingester/ring endpoint on distributors, ingesters will look healthy there.

Expected behavior

Ring page should show the same information when accessed via distributors and ingesters. In particular, ingesters should not use "heartbeat period" instead of "heartbeat timeout".

Additional Context

Original public Slack thread: https://grafana.slack.com/archives/C039863E8P7/p1653301187771079

@pstibrany pstibrany changed the title Ring view Ring view accessed via ingester shows unhealthy ingesters, even if they are healthy May 24, 2022
@pracucci pracucci added bug Something isn't working component/hash-ring good first issue Good for newcomers labels May 24, 2022
@zenador zenador self-assigned this May 25, 2022
@pracucci pracucci reopened this May 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working component/hash-ring good first issue Good for newcomers
Projects
None yet
3 participants