-
Notifications
You must be signed in to change notification settings - Fork 87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Managed Prometheus does not support _count exemplars #812
Comments
It's pretty unlikely we will support counter exemplars in the near term. In Monarch, exemplars are stored with the point itself, and only histograms have the necessary data structure (proto) for that. Counters and Gauges are stored far more efficiently and there's no place to put an exemplar. Changing this would likely require us building an entirely separate exemplar storage system, which isn't likely given that most of the need for exemplars is on histograms. That being said, we should probably give you a way to ingest these metrics without exemplars (dropping the exemplar + ingesting the time series, instead of erroring) so that you don't have to modify the exporter to get these metrics. I'll open a FR for this, thank you for letting us know! |
To answer this part of the question, we are currently deploying Prometheus v2.43 out to the GKE Rapid release channel for new clusters within the next few days. We believe the upgrade should suppress the error but the exemplars will be silently dropped, although we still need to test and verify this. |
I think we can live with exemplars being dropped in that case. It is much better from our point of view to drop the exemplars rather than failing to scrape and not getting any metrics at all.
Sounds great, I'm looking forward to it! Thank you for the quick responses! ⭐ |
@TheSpiritXIII Can you clarify please how to get the updated Managed Prometheus v2.43 for an existing GKE cluster? Is it enough to just upgrade the GKE cluster to the latest version from the GKE Rapid release channel using the usual GKE cluster upgrade menu in the Cloud Console? Will the Managed Prometheus instance be upgraded to v2.43+ automaticlly? And how to be notified when Managed Prometheus v2.43 will be available in the GKE Rapid release channel? Just to force the upgrade and check if it works. |
Hi @xak2000! If you just want to test it, creating a brand new cluster on the latest GKE minor version using the Rapid release channel will always have the latest GMP version. Existing clusters on the latest GKE minor version are being upgraded randomly in these next few weeks (outside of our control). Our latest version requires |
The cluster that I wanted to use to test it is on the But if the upgrade to the latest GMP version is not guaranteed with the switch of the release channel, then probably it's not worth it. Creating a new cluster, configuring the build environment etc. to deploy apps (with metrics exporters) that I want to test to the new cluster, etc. - could be too time-consuming. I'll wait until Thank you @TheSpiritXIII, this helped! |
Appending this here, in case anyone else see this 🧵. Set the following params entry in |
I'm a bit puzzled here since exemplars for I guess GKE managed Prometheus supports exemplars on Does anyone know why an unsupported version of Prometheus is used in GKE? 2.50.1 is the current Prometheus Server version. Does GKE offer its own Prometheus build with a separate support lifecycle? |
Regarding why we don't support Counter exemplars - this might not be a satisfying answer, but we intentionally don't aim to maintain 100% compatibility with OSS Prometheus. Our goal is to support the vast majority of use cases, while strategically compromising on some lesser-used use cases so that we can continue to use the power and scale of Monarch to back the system. Supporting every single Prometheus function would require building a backend from complete scratch, and doing that would preclude us from supporting other highly-requested non-Prometheus flows such as querying Cloud Monitoring metrics with PromQL and exporting data from managed Prom to BigQuery. We do aim to become more conformant over time, but unfortunately Counter exemplars is one of those features that would be prohibitively expensive for us to add due to how deep the incompatibility is baked into Monarch. Monarch supports exemplars on the individual histogram buckets, in this case it'd be the _bucket time series, which are only found on histogram metrics. Due to current architecture constraints, we need to maintain and redistribute a small fork of Prometheus to support managed collection. Given our scale and our customers' reliability needs, this requires us to do a lot of qualification of each new OSS release, meaning we don't support every new version of Prometheus. Our current policy is to support every LTS release plus new versions that substantially change functionality plus whatever kube-prometheus pegs to. Once OSS prometheus supports remote write v2 or OTLP, we likely will be able to sunset our fork, at which point we can peg to head much more easily and quickly. Hope this helps clarify. |
Thank you very much for the explanation! I would like to call out that the lack of this (the exclusive support of Histogram buckets) makes it hard to the users to troubleshoot anything else than latency-related issues. E.g.: an error rate went up, give me an example trace for that time series to see what happened. I might have diverted a conversion with this a little, sorry for my confusion. Let me go back to the original issue description. I think the title and the description of the issue might be misleading. Even if GCP's Prometheus fork does not support exemplars on So I think the solution for the original issue is not supporting exemplars for Does this make sense? |
+1 - agreed on your assessment, upon further review it looks like once 2.45 is rolled out this issue will fix itself automatically. We're still qualifying it... should be soon though. |
Also - to add to my previous statement about pegging to new versions - we are Fedramp compliant w.r.t. patching CVEs and vulnerabilities. This all happens behind the scenes. |
Is there any workaround until the fix is provided to drop the exemplars metrics and Prometheus can scrape other metrics from the target. metricRelabelings: # happens after the scrape
- action: drop # drop exemplars metrics not supported by GCP prometheus
regex: 'spring_kafka_(template|listener)_seconds_count|hikaricp_connections_(acquire|usage)_seconds_count'
sourceLabels: [ __name__ ]
relabelings: # happens before the scrape
- action: drop # drop exemplars metrics not supported by GCP prometheus
regex: 'spring_kafka_(template|listener)_seconds_count|hikaricp_connections_(acquire|usage)_seconds_count'
sourceLabels: [ __name__ ] |
@GitKaran hello! Prometheus |
Thanks @TheSpiritXIII I deployed |
This should be resolved now on all the latest patches of all supported GKE minor versions. Please let us know if you have further issues. Thanks! |
According to this page: https://cloud.google.com/stackdriver/docs/managed-prometheus/exemplars
There is no support for counter exemplars: "Exemplars attached to counter metrics can't be ingested"
The problem is that we are using spring-boot, and they have added exemplars to other types:
https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.2-Release-Notes#broader-exemplar-support-in-micrometer-112
as this is available for all supported prometheus version.
This leads to us not being able to scrape any metrics when using managed prometheus.
Port-forwarding to collectors pods, and checking the GUI, shows errors like this:
Is there any timeline for when managed prometheus will update to prometheus v 2.43 or later, and start supporting exemplars for other types than histograms?
This is currently preventing us from fully migrating away from our own prometheus to the managed one.
The text was updated successfully, but these errors were encountered: