Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cluster: Add the option to set a memberlist label for the cluster #44

Closed
wants to merge 3 commits into from

Conversation

JohnnyQQQQ
Copy link
Member

This PR adds the option to set the Memberlist label for the created cluster. This can be important if multiple clusters are run i.e. on Kubernetes, and IPs get recycled. Some more context from this blogpost:

Why memberlist labels matter
We would be doing our community a disservice if we didn’t mention the importance of using labels when using embedded memberlist.

In Grafana Cloud, for the sake of simplicity (and in order to run as few open ports as we can), Mimir/Tempo/Loki all run memberlist using the same service name, Kubernetes label, and port. Mostly.

Because of the way Kubernetes works, it is possible for it to reassign the same IP address to pods in different namespaces when pods stop/start. When using the same Kubernetes service name and port, this can result in a pod within a Loki namespace communicating with a pod in the Mimir namespace whose IP previously belonged to a Loki pod within Loki’s ring. This problem can be mitigated with the use of labels in memberlist, ensuring that cross namespace memberlist communication is rejected by the processes running memberlist themselves.

Once you add labels to memberlist gossip messages, each component in a namespace’s ring will not accept messages that don’t have the required label. We set this label config value to k8s cluster.k8s namespace to ensure it is unique across clusters and namespaces.

If you are migrating to memberlist from an existing KV store, you can simply add the configuration to set your memberlist label in step 1 of the earlier migration steps. However, if you are already running memberlist and plan to deploy another Loki instance (or any other database) in the same IP address space you should migrate your existing deployment to use labels.

santihernandezc and others added 2 commits April 26, 2023 15:25
This PR makes it possible to store the last error for each receiver in case of notification delivery failure. These errors are exposed via the `/api/v2/receivers` endpoint.

Co-authored-by: gotjosh <josue.abreu@gmail.com>
Capture and expose notification delivery errors (#31)
@JohnnyQQQQ JohnnyQQQQ added the enhancement New feature or request label May 5, 2023
@JohnnyQQQQ JohnnyQQQQ force-pushed the add-memberlist-label-option branch 2 times, most recently from cbd515d to 846e1e2 Compare May 5, 2023 09:57
@JohnnyQQQQ JohnnyQQQQ force-pushed the add-memberlist-label-option branch from 846e1e2 to dde1f05 Compare May 5, 2023 09:58
@JohnnyQQQQ JohnnyQQQQ requested a review from gotjosh May 5, 2023 10:34
@gotjosh
Copy link
Collaborator

gotjosh commented May 5, 2023

Shall we upstream this directly? Then we can just sync the fork pretty quickly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
4 participants