Docs: Add Documentation to do HA deduplication on Mimir Helm Deployment

[lamida]

What this PR does

Add Documentation to do HA deduplication on Mimir Helm Deployment.

Which issue(s) this PR fixes or relates to

Fixes #2597

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[krajorama]

Good start! In general I think we need to add Grafana agent setup and assume that the user already has a working Mimir - to avoid duplicating documentation.

[lamida]

@krajorama please help to do another pass when you are free. I will unset draft from this PR too now.

Changes from your first pass:

• Removed install mimir step and make mimir installated as prerequisite
• Because reference to install mimir is removed therefore I don't see it makes sense to keep Prometheus helm install guide too
• Removed grafana port forwarding and data source setup
• Added Grafana agent setup (although the config and setup are exactly same like Prometheus)

[dimitarvdimitrov]

thanks for writing this! I would also add a mention to this article in the existing "Configuring Grafana Mimir high-availability deduplication"

[dimitarvdimitrov]

nit: I'd also add a link to the HA dedup docs here, since here is the first mention of HA dedup in this doc.

[dimitarvdimitrov]

This is covered by the other guide. Should we repeat the info here too?

[lamida]

I am worried that keep referring to the external document will make the reader keep jumping around to different documents. External label configuration is the most important part of HA configuration, hence I feel that for good reading/guide continuity, it will be better to keep this here.

[osg-grafana]

• We need to add a note about what content is copied and pasted between documents so that we name one source as definitive. This is a workaround because we do not have single-sourcing capabilities for documentation that is this granular.

[lamida]

I decided to put a note that this section contains similar content from another HA doc. Please review and advise if this note is useful or better just to remove it.

[dimitarvdimitrov]

Can we use the name of the chart - mimir-distributed? Maybe a tech writer can also pitch in, but using the name seems more concrete and searchable

[osg-grafana]

I will take a look at this soon; on UTC-4 atm and might not get to this until next Monday.

[dimitarvdimitrov]

I haven't tried installing consul. Is there anything specific about this installation that they should do? Like configure this as X or the other thing as [A, B, C]? or the default config would work?

[lamida]

Default config will work.

[dimitarvdimitrov]

nitpick about values.yaml. I think we've used custom.yaml in a few articles so far to refer to the values file that the user supplies. It makes it obvious that it's a different file from the values.yaml in the root of the helm chart. Those values.yaml are the default ones and helm uses them every time, even without providing a flag.

[dimitarvdimitrov]

the distributor also exposes some metrics around this. I think it's worth mentioning. They all start with cortex_ha_tracker_, I think the ones that start with cortex_ha_tracker_elected_replica_ will be most useful to users.

These are also exposed on the "Mimir / Tenants" dashboard in the "Distributor deduplicated/non-HA" panel.

[dimitarvdimitrov]

on another topic - what do you think about moving this section to the other article - "Configuring Grafana Mimir high-availability deduplication"?

[lamida]

the distributor also exposes some metrics around this...
I will mention the metrics info in this section.

on another topic - what do you think about moving this section to the other article - "Configuring Grafana Mimir high-availability deduplication"?

I am fine to move the section to the "Configuring Grafana Mimir high-availability deduplication". But maybe we want to hear first what @krajorama or @osg-grafana think.

[krajorama]

Mimir does not have self monitoring like GEM so that won't make sense there in itself.
And the Mimir documentation tries to be deployment agnostic so talking about port forward seems out of scope.
Given that I'd keep these here.

[osg-grafana]

Change this alias and title to use an imperative: configure-helm-ha-deduplication-consul.

[osg-grafana]

Does this mean, "Put other configurations before this one." or something else?

[lamida]

I mean the section above this comment is all configurations added when installing mimir-distributed helm chart in the first time.

[lamida]

Anyway to make it non confusing, I just removed the whole comment.

[osg-grafana]

Suggested change

	Upgrade the Mimir's helm release using the above configuration.
	1. Upgrade the Mimir’s helm chart version by using the following configuration.

This looks inaccurate, so I changed it to "following".

[lamida]

The "following" code block is not a configuration but is a command. Hence I changed this to:

2. Upgrade the Mimir's helm release using the following command:

[osg-grafana]

1. Upgrade the Mimir’s helm chart version by using the following configuration:

[osg-grafana]

The image docs/sources/operators-guide/configure/setting-helm-ha-deduplication-consul/verify-deduplication.png is unreadable. Do you have one that has higher readability?

[pracucci]

The CHANGELOG has just been cut to prepare for the next Mimir release. Please rebase main and eventually move the CHANGELOG entry added / updated in this PR to the top of the CHANGELOG document. Thanks!

[osg-grafana]

Suggested change

	Configure Prometheus or Grafana Agent HA setup by setting label called `cluster` and `__replica__`. These two labels
	name are default labels for HA setup in Grafana Mimir. If you want to change the labels, make sure to change it in Mimir
	too to make Grafana Mimir and Prometheus/Grafana Agent config matches with each other, otherwise HA deduplication
	wouldn't work. `cluster` label value must be same across replica belong to the same cluster. `__replica__` must be
	unique across different replica in the cluster.
	Configure the Prometheus or Grafana Agent HA setup by setting the labels named `cluster` and `__replica__`,
	which are the default labels for a HA setup in Grafana Mimir. If you want to change a label,
	make sure to change it in Mimir as well. This ensures that the configurations of Grafana Mimir, Prometheus, and Grafana Agent all match each other. Otherwise, HA deduplication will not work.
	* The value of the `cluster` label must be same across replica that belong to the same cluster.
	* The value of the `__replica__` label must be unique across different replica within the same cluster.

[osg-grafana]

Please vet this for technical accuracy.

[lamida]

For technical accuracy, the second sentence should be something like

If you want to change the HA labels, make sure to change them in Mimir as well.

instead of

If you want to change a label, make sure to change it in Mimir as well.

because we are talking about specific label pairs with the default name cluster and __replica__.

Please review it again.

[osg-grafana]

Suggested change

	2. Upgrade the Mimir's helm release using the following command:
	2. Upgrade the Mimir's Helm chart release using the following command:

[osg-grafana]

Is this Helm or the Helm chart? Please vet my change for technical accuracy.

[lamida]

We are upgrading mimir-distributed helm chart release. Since we are upgrading the mimir-distributed chart using the commands that follow that line, a new release will be installed.

See this reference

[lamida]

We are upgrading mimir-distributed helm chart release. Since we are upgrading the mimir-distributed chart using the commands that follow that line, a new release will be installed.

See this reference

[krajorama]

This is looking good to me now overall. I wonder where to add some information about using sharding in prometheus operator that I discovered during a support case: https://github.com/grafana/support-escalations/issues/4030#issuecomment-1262289642
I'm fine to add it in a separate PR though after we merge this.

[krajorama]

With two nitpicks

[lamida]

This is looking good to me now overall. I wonder where to add some information about using sharding in prometheus operator that I discovered during a support case: https://github.com/grafana/support-escalations/issues/4030#issuecomment-1262289642
I'm fine to add it in a separate PR though after we merge this.

Yes most likely it is better to be covered in separate PR. I am still more than happy to work on that.

[krajorama]

Please rebase to current main and move this line to around line 15 in the main/unreleased section

[lamida]

Sorry missed to do it immediately yesterday. Updated now.