Federating: any way of disabling smartness of adding instance/job entirely?

[EdSchouten]

We want to configure a set of Prometheus instances to federate for multiple purposes:

• Running Prometheus with short-term retention for the bulk of metrics,
• Provide separate Prometheus instances for long-term retention for selected data.
• Provide separate Prometheus instances that only contain data for a given customer, so that customers can explore the data freely.

What we've observed is that Prometheus doesn't seem to provide an easy way of scraping another Prometheus instance and leaving all of the labels in its original form. It always tries to add 'instance' and 'job'. The 'honor_labels' flag doesn't help here entirely, as it only determines how to deal with conflicts -- not when one of those labels was absent on the original dataset. We have some aggregated rules that don't have job/instance set, meaning these labels still get added when federating. Right now we're working around by adding this:

  # Set instance and job to placeholder markers before scraping.
  relabel_configs:
  - replacement: quxquxqux
    target_label: instance
  - target_label: job
  # Remove the placeholders if still present.
  metric_relabel_configs:
  - replacement: ''
    regex: quxquxqux
    source_labels: [instance]
    target_label: instance

Unfortunately, this approach has the downside of also partially messing up built-in metrics like up, scrape_duration_seconds, etc.

My question is, are there any plans on improving this in future versions of Prometheus? Are there any proposals for the desired way of doing this? Is there any way we can help out?

[brian-brazil]

Honor_labels should be sufficient here to take care of the instance label, if not it's a bug in the federation endpoint.

There should always be a job label though, you should ensure your aggregations are preserving it.

[EdSchouten]

The problem with honor_labels is that it doesn't take a look at the entire dataset it scrapes. It is considered per timeseries. This means that if I'm aggregating timeseries and thereby discarding the instance field on one Prometheus instance, it is added back on the other Prometheus instances that is federating, containing the hostname of the former. For example, if I'm writing a rule like this that counts the total number of http requests of simply everything running in a single datacenter:

datacenter:number_of_http_requests:rate5m = sum(number_of_http_requests) by (datacenter)

When pulled in through federation, it suddenly has an instance label, which is clearly not what I'd expect. I wouldn't mind if it used any other label (e.g., federating_instance), but reusing instance just looks silly.

To reply to your final remark: why should there always be a job label? What if I'm simply interested in summing up all metrics of a given name, regardless of the name of the job? It shouldn't be Prometheus' business to care about that.

Prometheus would have been more functional if there was a config option add_implicit_labels which I could set to false.

[brian-brazil]

To reply to your final remark: why should there always be a job label?

So you can distinguish metrics from different jobs.

What if I'm simply interested in summing up all metrics of a given name, regardless of the name of the job?

This doesn't come up too often, but I've seen a job label such as "cluster" used in this case.

Prometheus would have been more functional if there was a config option add_implicit_labels which I could set to false.

This is what honor_labels is intended for.

[brian-brazil]

This is #2488

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.