Helm: multitenancy by default makes migrations to mimir more difficult

[dimitarvdimitrov]

This has come up from a few members in the community slack since the 3.0 release (example1, example2). So I'm trying to summarize it in an issue.

Context

Deploying mimir via the helm chart. Use case is for migrating from a cortex cluster without multitenancy to a mimir cluster with multitenancy. I want to redirect my old clients to the new cluster and still ingest their metrics even though they aren't providing the x-scope-orgID header. I also want to allow new tenants to send metrics to mimir with their own tenant IDs.

Problem

With multitenancy enabled by default in the chart Mimir can only ingest metrics without x-scope-orgid header as the anonymous tenant. While the option is configurable in mimir, the nginx in the chart sets a default header value. There are workarounds:

Proposal

The chart should automatically detect my mimir config and the no_auth_tenant there and use that in the nginx config.

Alternative 1

Rename my fake tenant in cortex and then move it to mimir. Theoretically this only requires changing the name of the tenant in the folders/prefixes in the object storage.

But in reality with a running system this requires downtime. Ingestion, compaction and rule evaluation should stop, blocks should be flushed to object storage and then I can rename my tenant to anonymous in the object storage.

Alternative 2

Copy nginx.config from the chart and change the default value for the x-scope-orgid header that nginx adds from anonymous to fake. This means that I have to now keep my nginx config up to date with the one in the chart.

Alternative 3

Change my clients to now pass the correct header. Maybe doable, despite requirements, but this may be difficult to do if not all clients are managed by me.

NB all of this applies for when someone was running without multitenancy with the mimir-distributed chart and had a non-default no-auth tenant and wants to now upgrade to version 3.0 of the chart

[krajorama]

agree with the proposal. mimir config is extremely unlikely to include the nginx config, so should not run the risk of cyclic dependency :)

[osg-grafana]

Would this issue require (internal?) documentation?

[krajorama]

Yes, I think we'll need to update the migration guide from cortex and not just because of this issue.