You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I understand that AWX is open source software provided for free and that I might not receive a timely response.
I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)
Bug Summary
Starting with version 2.52.0, Prometheus introduced a mechanism to detect duplicate series during scraping. This can lead to error logs when awx scrapes metrics.
prometheus debug logs:
ts=2024-05-29T13:39:18.501Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/awx-host-metrics-monitor/0 target=http://10.42.1.111:8052/api/v2/metrics msg="Duplicate sample for timestamp" series="subsystem_metrics_pipe_execute_seconds{node=\"awx-task-5969f8b88c-dlpnj\"}"
ts=2024-05-29T13:40:33.500Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/awx-host-metrics-monitor/0 target=http://10.42.1.111:8052/api/v2/metrics msg="Duplicate sample for timestamp" series="subsystem_metrics_send_metrics_seconds{node=\"awx-task-5c4cdcf96d-9j9wm\"}"
ts=2024-05-29T13:41:03.511Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/awx-host-metrics-monitor/0 target=http://10.42.1.111:8052/api/v2/metrics msg="Duplicate sample for timestamp" series="subsystem_metrics_pipe_execute_calls{node=\"awx-task-5969f8b88c-dlpnj\"}"
"ts=2024-05-29T13:54:43.501Z caller=scrape.go:1738 level=warn component=\"scrape manager\" scrape_pool=serviceMonitor/monitoring/awx-host-metrics-monitor/0 target=http://10.42.1.111:8052/api/v2/metrics msg=\"Error on ingesting samples with different value but same timestamp\" num_dropped=9"
Kubernetes version: 1.28.10+rke2r1
AWX version
24.4.0
Select the relevant components
UI
UI (tech preview)
API
Docs
Collection
CLI
Other
Installation method
kubernetes
Modifications
no
Ansible version
No response
Operating system
Ubuntu 22.04.4
Web browser
Chrome
Steps to reproduce
On a fresh rke2 installation:
deploy prometheus operator from kube-prometheus-stack enable logLevel:debug
deploy AWX with operator deploy service monitor to scrape metrics
Expected results
There should not be any error regards duplicate metrics
Please confirm the following
security@ansible.com
instead.)Bug Summary
Starting with version 2.52.0, Prometheus introduced a mechanism to detect duplicate series during scraping. This can lead to error logs when awx scrapes metrics.
prometheus debug logs:
Kubernetes version: 1.28.10+rke2r1
AWX version
24.4.0
Select the relevant components
Installation method
kubernetes
Modifications
no
Ansible version
No response
Operating system
Ubuntu 22.04.4
Web browser
Chrome
Steps to reproduce
On a fresh rke2 installation:
Expected results
There should not be any error regards duplicate metrics
Actual results
The one described in the logs
Additional information
Already discussed here on why Prometheus changed behaviour prometheus/prometheus#14089
The text was updated successfully, but these errors were encountered: