Automatically include `node` label for every `instance`

[SuperQ]

It's highly useful to be able to metrics comparisons between targets on the same node. For example getting the rate of disk IO from the node_exporter and normalizing that with request counters from mysqld_exporter.

Creating a node label from instance works via metrics relabeling, but it can be very messy. It would be nice to just have this be a default option.

[brian-brazil]

This is the sort of thing that's going to vary a lot between sites, and not going to work if you've things like an overlay network or multiple DNS names for a given machine in play. As with all cases where you're trying to add labels to targets that are site specific, relabelling is the way to go.

[SuperQ]

Except that a node is a very common labeling method for a lot of network designs. Even in something like kubernetes/docker where you have per-container IP address, you may have multiple targets that have relevant metrics to share with each other. Forcing the node:port as the instance just makes it more difficult to aggregate and combine metrics. These should never have been a single label to begin with. instance_node and instance_port would have been better.

[brian-brazil]

I think you're confusing __address__ and instance. While __address__ can currently only be a host:port (and it's conceivable we'll have unix domain sockets some day), there's no such restriction on instance. It could be an ip:port, a dnsname:port, a dnsname, an IP, one of the various formattings of IPv6 addresses, a URL, a zookeeper path, a consul path, a EC2 instance ID or anything else that makes sense in the context of your deployment and in particular your service discovery.

This is something that's only make sense in some deployments, and adding in labels like this will cause problems for users as it'll require additional special handling by users to strip off as we improve label handling in promql.

This is not something we can safely do by default due to both safety, and because it's not sane from a maintenance/usability standpoint to offer options, let alone defaults, for the multitude of common scenarios.

[SuperQ]

How about a global relabel section in the config so I don't have to repeat the relabel for every job?

[brian-brazil]

Hmm, I can see that working in some cases but not working in most cases when you consider things like the blackbox exporter which require different handling. In general I'd also expect the node exporter and applications to have different relabel rules. I think this is probably better handled by a for loop in configuration management, as there's likely lots of other aspects that are similar if relabelling is.

[fabxc]

I agree here. This requires a single relabeling rule – way less than needed for other tweaks to the standard behavior.
Overly noisy repetition generated by such loops can often be avoided by making better use of the existing SD capabilities.

Relabeling is not trivial in itself, layering it will make configuration unnecessarily complicated.

[fabxc]

Closing since we solved this accordingly for our use case.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.