-
Notifications
You must be signed in to change notification settings - Fork 8.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prometheus v2.52.0 raises "Error on ingesting samples with different value but same timestamp" for kube-state-metrics #14089
Comments
I might has something to do with bairhys/prometheus-frigate-exporter#9 and the introduced check for duplicated series #12933 |
Yes, starting with Line 1625 in 3b8b577
cc @bboreham as you reviewed the feature. |
@machine424 I already thought that, but I don't see any duplicates in the metrics. So I'm a bit confused right now. However, I like the idea of showing the failing metrics in the debug log. |
Yes, it's not that easy to debug. If you want to add that log, please go ahead. We'll see if we can add it to any potential |
I see this from the report:
This is intentionally not giving any details on series, just the number. |
This makes it harder to tell if your problem could be relabeling. |
Actually, now that I'm looking at the code for real, I think a debug log should already be provided via Lines 1781 to 1785 in 3b8b577
(no need for the extra debug log) You can --log.level=debug and see.
|
@machine424 you are right the debug log is already implemented. I just deployed one prometheus with debug log level enabled. Seems like kube-state-metrics is indeed producing duplicate samples...
|
And well it is doing by purpose... not sure what AKS is doing here but the toleration exists two times on the calico-kube-controllers deployment
So it seems like everything is working fine on prometheus and kube-state-metrics side 👍 |
“Ignored” is probably the wrong word here. It’s a little bit more complicated than that. |
I think this is worth creating an issue on |
@machine424 will do. Quite confusing to see that "duplicates" are allowed within the |
Closing here - Since everything is working as expected with Prometheus, thanks to everyone your help! |
For those finding this issue and wanting to follow on with kube-state-metrics, you want: |
Thanks for investigation @machine424. |
Good point. I think we agree that even in such cases (same value), we should continue to consider it as an error. This can help highlight a hidden issue (targets shouldn't rely on Prometheus deduplicating that IIUC). But I'm afraid some targets may be relying on the old behavior, especially the ones with That being said, the TSDB doesn't consider samples with the same timestamps and the same value as duplicates; it tolerates that: prometheus/tsdb/head_append.go Lines 464 to 473 in dc92652
Hence, the explicit warning message. If we want to maintain the current behavior, I agree we shouldn't use a storage error |
I'd also would like to suggest a revision of the warning message triggered by duplicate series in Prometheus. In my experience, the message didn't accurately reflect the situation, as both samples had identical values. Similarly, the While I understand the logic behind rejecting duplicate samples, I'm a bit confused about the implementation, as the underlying TSDB is accepting such cases. |
Duplicate labels on scrape is a clear logic error (at least in the mind of some people who worked on it). Duplicate sample fed into TSDB is something that happens, e.g. on some kinds of restart, and we prefer simple logic to always accept it over complicated logic aimed at particular corner cases. I only wanted to nitpick the wording of the message, not change behaviour. See also #13277 (comment). Unfortunately it would be more of a breaking change to rename |
What did you do?
Hello,
With the update to Prometheus v2.52.0 (https://github.com/prometheus/prometheus/releases/tag/v2.52.0), an error has been introduced in Prometheus logging, indicating duplicated samples from kube-state-metrics.
Consequently, it has triggered the following rule, which is part of the Prometheus Operator's kube-prometheus (https://github.com/prometheus-operator/kube-prometheus) project, which I use for deploying the monitoring environment:
I don't see any duplicates in these metrics, which raises the question of why the scrape manager is reporting an issue.
kube-state-metric
/metrics
What did you expect to see?
No response
What did you see instead? Under which circumstances?
see logs area
System information
No response
Prometheus version
Prometheus configuration file
No response
Alertmanager version
No response
Alertmanager configuration file
No response
Logs
The text was updated successfully, but these errors were encountered: