Remote write relabelling

[brian-brazil]

We should support relabelling on the remote write path. This goes after external labels.

[juliusv]

@brian-brazil Could you add the rationale for why this goes after external labels, especially since it's the other way around with alerts? Naively I would have thought that one wouldn't want to let the user shoot down the external labels, but who knows...

[brian-brazil]

The external labels logic is just a default, it needs to be overrideable (especially for alerts). If alert relabelling is the wrong way, around that's a bug.

[juliusv]

@brian-brazil Ok, then maybe we should change the alert relabeling order before documenting it - can we get around calling it a breaking change then? :)

[juliusv]

Of course, technically you can achieve the same either way by just not using external labels for any labels that you may want to override (and just use relabeling to set them everywhere). It's just not as convenient. But it gives you the guarantee that anything outgoing from a given Prometheus server will always have the configured external labels.

[brian-brazil]

If external labels is 2nd, you can't set an external label to empty. External labels are also defaults, not overrides - if you want to make them overrides use relabelling.

[juliusv]

Don't care much either way - question is whether we consider it breaking to "fix" the way it's done in alerting currently.

[brian-brazil]

The alert relabelling feature as implemented doesn't meet the spec of the issue requesting it, so that's a bug.

[juliusv]

Phew 😅

For reference: #1744

[brian-brazil]

Also dupe of #1955

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.