Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update -memberlist.join flag to use DNS-SD format #2549

Merged
merged 6 commits into from
Jul 26, 2022

Conversation

dimitarvdimitrov
Copy link
Contributor

@dimitarvdimitrov dimitarvdimitrov commented Jul 26, 2022

What this PR does

The default value for -memberlist.join in jsonnet and helm does not use DNS service-discovery format. Without using this format, the change in grafana/dskit#195 will not result in fast joining. The resolution in dskit will not resolve the -memberlist.join address. Instead, resolution will be left to the memberlist library. Memberlist will resolve to pod IPs and will effectively join all nodes during the supposed "fast-join" phase.

Which issue(s) this PR fixes or relates to

Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Copy link
Member

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for spotting the problem. I confirm that dns+... format helps memberlist KV's fast-join operation to work correctly.

Correct output is:

level=info ts=2022-07-26T14:58:42.373220881Z caller=memberlist_client.go:545 msg="memberlist fast-join starting" nodes_found=37 to_join=8
...
level=info ts=2022-07-26T14:58:42.45983464Z caller=memberlist_client.go:565 msg="memberlist fast-join finished" joined_nodes=8 elapsed_time=102.926227ms

(nodes_found=37 and joined_nodes=8 is correct for "fast-join")

Previously wrong output:

level=info ts=2022-07-25T13:42:29.567761114Z caller=memberlist_client.go:545 msg="memberlist fast-join starting" nodes_found=1 to_join=4
...
level=info ts=2022-07-25T13:42:29.913222231Z caller=memberlist_client.go:565 msg="memberlist fast-join finished" joined_nodes=34 elapsed_time=345.466911ms

(nodes_found=1 and joined_nodes=34 is wrong)

CHANGELOG.md Outdated Show resolved Hide resolved
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
@pstibrany pstibrany enabled auto-merge (squash) July 26, 2022 15:17
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
@pstibrany pstibrany merged commit a943542 into main Jul 26, 2022
@pstibrany pstibrany deleted the dimitar/memberlist-join-addr branch July 26, 2022 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants