Add a webpage for target relabeling debug

[valyala]

Is your feature request related to a problem? Please describe.

Sometimes target relableing may lead to unexpected results. For example:

• the target gets unexpected labels after applying the relabeling
• the target is unexpectedly dropped after applying the relabling

It is hard to detect the relabeling rule or the set of labels at the discovered target, which could lead to such unexpected results.

Describe the solution you'd like

It would be great if vmagent and single-node VictoriaMetrics could provide a special webpage for target relabel debugging at /target-relabel-debug?target_id=... path.

This webpage should contain information, which should simplify answering the following questions:

• why the target has been dropped during the relabeling?
• why the target has the given label values after the relabeling?
• which particular relabeling rule led to target drop?
• which particular label at the discovered target led to unexpected relabeling?

Given these questions, the webpage should contain a list of per-relabeling rule entries with the following information:

• the relabeling rule
• the list of target labels before applying the relabeling rule
• the list of target labels after applying the relabeling rule

Every target entry at /targets page and at /service-discovery page must contain a link to
/target-relabel-debug?target_id=... webpage, so users could click on the link and instantly understand why the particular target was dropped or why the target has unexpected labels.

Describe alternatives you've considered

An alternative is to manually apply relabeling rules to the original target labels listed at the /service-discovery page via third-party service such as relabeler playground. Unfortunately this service doesn't support relabeling enhancements implemented in vmagent.

Another alternative is to enable debug logging for relabeling at vmagent according to these docs. This solution isn't trivial to use. It also doesn't allow identifying the particular relabeling rule, which led to unexpected results.

Additional context

Related issues:

• vmagent: debug relabeling #1343
• vmagent: there is no (easy?) way to find out why certain target was dropped #3311

[valyala]

The commit a8b8e23 substitutes the old-style debugging for relabeling via relabel_debug and metric_relabel_debug options with the enhanced debugging capabilities described in these docs. This commit will be included in the next release. In the mean time it is possible to build vmagent from this commit accroding to these docs. Another option is to play with metric-level debugging built into vmselect at VictoriaMetrics playground at https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/metric-relabel-debug .

[valyala]

The relabel debug is available starting from VictoriaMetrics v1.85.0 - see these docs. Closing the feature request as done.