Runtime override of tenant-specific active series custom trackers

[gubjanos]

What this PR does

This PR moves the active series custom trackers to a runtime configuration, and also introduces tenant-specific overwrites for these matchers.
I am uploading this to start an early discussion of the design,

Key points I would already like to discuss:

• Moving the custom trackers to runtime configuration is a breaking change as it removes the flag and changes the default behavior
• Config change management could be better handled on manager side, with some content hashing, but that would introduce dskit changes...

Which issue(s) this PR fixes

https://github.com/grafana/mimir-squad/issues/526

Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[colega]

I like the direction where this is going, however before going into a deeper review about nitpicks etc. I'd like to discuss the fact that per-tenant matchers overwrite the default ones.

IMO, the per-tenant matchers should be merged on top of the default ones. We could leave the door open to un-setting a matcher if needed (weird need, but who knows).

Consider that we have only one default matcher:

• integration/caddy:...

And we set per-tenant matchers for a tenant:

• integrations/caddy:... (we need to carry this default one)
• team_a_metrics:...

After that, a system operator adds a new default one integration/k8s, but now they need to update all the per-tenant ones too, otherwise they'll get outdated.

[gubjanos]

Actually, I have started the implementation in the way that you have just described.
I have decided to implement without merging because:

• Based on our current configurations in jsonnet, it is easy to include the default set of matchers in tenant based configurations and it is not that hard to maintain. So the main idea was to keep it as simple as possible on ingester side, and offload the complexity to configuration
• If we implement the default matchers + tenant-specific ones, we need to solve a complex config diffing problem (we need to check if the defaults change + the user specific ones) which result in more complex logic in the ingester
• It is also not clear how to handle when the defaults and the user specific matchers have overlap

But all in all, I can easily be convinced to do the other way. :)

[colega]

Oh I see, so you still want to merge but you want to do that in our jsonnet.

[gubjanos]

One detail that's disturbing me a bit is terminology: technically "matcher" is a label="value" part (more correctly called "label matcher"), while {label="value", another=~"aaa.*"} is a selector. It would be a nice to fix the terminology, but it may be too late for that.

I agree with that. We have agreed to work on that in separate PR to limit the diff size of this one.

[CLAassistant]

All committers have signed the CLA.

[pstibrany]

Please rebase this PR on top of main to make all required test checks pass.

[colega]

This looks good to me! Left a minor nitpick. Not leaving a green tick because as I mentioned I shouldn't be approving this since I collaborated on it.

[pstibrany]

Thank you. This is massive PR, but you did an amazing job navigating unfamiliar Mimir code and our comments and suggestions. Great job!

(I left few remaining small comments, but nothing blocking.)

[pstibrany]

Thank you. This is massive PR, but you did an amazing job navigating unfamiliar Mimir code and our comments and suggestions. Great job!

I forgot: @colega also did an amazing job on this PR. It would not be this far without your help!

One detail: there is comment about removing deprecated option in Mimir 2.2. I think it will need to be in 2.3, because this feature will be released in Mimir 2.1.

[pstibrany]

Thank you for last touches on this!