Duplicate metrics causing issue with prometheus >= 2.52.0

[ilbarone87]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

Starting with version 2.52.0, Prometheus introduced a mechanism to detect duplicate series during scraping. This can lead to error logs when awx scrapes metrics.

prometheus debug logs:

ts=2024-05-29T13:39:18.501Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/awx-host-metrics-monitor/0 target=http://10.42.1.111:8052/api/v2/metrics msg="Duplicate sample for timestamp" series="subsystem_metrics_pipe_execute_seconds{node=\"awx-task-5969f8b88c-dlpnj\"}"
ts=2024-05-29T13:40:33.500Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/awx-host-metrics-monitor/0 target=http://10.42.1.111:8052/api/v2/metrics msg="Duplicate sample for timestamp" series="subsystem_metrics_send_metrics_seconds{node=\"awx-task-5c4cdcf96d-9j9wm\"}"
ts=2024-05-29T13:41:03.511Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/awx-host-metrics-monitor/0 target=http://10.42.1.111:8052/api/v2/metrics msg="Duplicate sample for timestamp" series="subsystem_metrics_pipe_execute_calls{node=\"awx-task-5969f8b88c-dlpnj\"}"
"ts=2024-05-29T13:54:43.501Z caller=scrape.go:1738 level=warn component=\"scrape manager\" scrape_pool=serviceMonitor/monitoring/awx-host-metrics-monitor/0 target=http://10.42.1.111:8052/api/v2/metrics msg=\"Error on ingesting samples with different value but same timestamp\" num_dropped=9"

Kubernetes version: 1.28.10+rke2r1

AWX version

24.4.0

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

Ubuntu 22.04.4

Web browser

Chrome

Steps to reproduce

On a fresh rke2 installation:

• deploy prometheus operator from kube-prometheus-stack enable logLevel:debug
• deploy AWX with operator deploy service monitor to scrape metrics

Expected results

There should not be any error regards duplicate metrics

Actual results

The one described in the logs

Additional information

Already discussed here on why Prometheus changed behaviour prometheus/prometheus#14089

[matburt]

dupe of #15179