Federation returns duplicate timeseries when matchers

[hdost]

What did you do?

Use /federate?match[]={__name__%3D~"(probe_.*)"}&match[]={job%3D~"http-local"}

What did you expect to see?

As mentioned in the documentation.

If multiple match[] parameters are provided, the union of all matched series is selected.

What did you see instead? Under which circumstances?

A "UNION ALL" to use SQL terminology.

Example:

probe_dns_lookup_time_seconds{dc="A",instance="https://a.example.com/health",job="http-local"} 0.001450639 1622622019358
...
probe_dns_lookup_time_seconds{dc="A",instance="https://a.example.com/health",job="http-local"} 0.001450639 1622622019358

Environment

• System information: Centos 7 ( This is a "business logic issue, doesn't really matter)

• Prometheus version:

2.27.1

• Alertmanager version: N/A

• Prometheus configuration file: N/A

• Alertmanager configuration file: N/A

• Logs:
  On the prometheus which is federating the out-of-order samples message is the symptom

[roidelapluie]

Thank you. This is indeed a bug. I recommend you the following workaround for the time being:

/federate?match[]={__name__%3D~"(probe_.*)"}&match[]={job%3D~"http-local",__name__!%3D~"(probe_.*)"}

[hdost]

Yup, I just wanted to make sure it was actually considered a bug 👍🏼

[roidelapluie]

It is! :)

[Harkishen-Singh]

Sorry, I didn't understand the bug here. Is it the repetition of series or the out of order samples log? (My guess is the first part).

[roidelapluie]

/federate should not return duplicate series

[Harkishen-Singh]

I can solve this as I don't think anyone else is working on it.