Prometheus Receiver - honor_labels set to true doesnt work with federation

[rashmichandrashekar]

Describe the bug
When using federation, honor_labels set to true doesnt work since the source prometheus server's job name is not found in the list of targets in the destination prometheus config

Steps to reproduce
Using federation, try to scrape metrics from another prometheus server with honor_labels set to true.

What did you expect to see?
Metrics collected correctly by the otel collector with the labels from the source prometheus server honored

What did you see instead?
An error that says "job or instance cannot be found from labels"

What version did you use?
Version: v0.27.0

What config did you use?
Config: (e.g. the yaml config file)

global:
  evaluation_interval: 5s
  scrape_interval: 5s
scrape_configs:
- job_name: federate
  honor_labels: true
  metrics_path: '/federate'
  params:
    'match[]': ['job:my_job_name']
  static_configs:
    - targets:
      - 'prometheus-prometheus-oper-prometheus.monitoring:9090'

Environment
OS: Ubuntu 20.04
Compiler(if manually compiled): go 1.14

Additional context
This seems to be related to the way targets are looked up based on job/instance labels. Related to #5663

[rashmichandrashekar]

cc : @dashpole - this seems to be related to the same issue as - #5663

[edude03]

Running into the same issue, is anyone facing this aware of a workaround?

[alolita]

@dashpole are you looking at this issue? We can also help so let me know.

[dashpole]

I am not currently working on it, but am happy to help if someone else wants to take it on

[Aneurysm9]

AFAICT, the issue is that the metric metadata is associated only with target labels, prior to merging them with individual metric labels, and the receiver, when it is given a metric to append, doesn't have access to the original value of instance and job if they have been overridden by labels in the exposition that are honored.

I think we might be able to do something with automatically appending relabeling rules that would put the original target job and instance label values in a new label with a name constructed to avoid conflicts. We could then use that value to look up the metadata and drop the label when converting to pdata. This might run into issues if label limits are set up, though.

[dashpole]

There is a different google project that uses a fork of the prometheus server. They dealt with this by embedding a metadata fetcher in the context passed to Appender (which be here in the prom receiver). I wonder if upstream prometheus maintainers would be amenable to such a solution...

[gouthamve]

I've run into this while trying to test my fix for #8355 and sent an upstream PR: prometheus/prometheus#10450

If the upstream PR is merged it should remove a lot of complexity from the collector codebase :)

[Utwo]

I think this issue is still happening on helm version 0.60.0

[shacharSirotkin]

I think this issue is still happening on helm version 0.60.0

Hii, did you find a workaround?

[Utwo]

I think this issue is still happening on helm version 0.60.0

Hii, did you find a workaround?

No, not yet. I'm using open-telemetry exclusively for collecting traces. For metrics, I moved to a different solution.