-
Notifications
You must be signed in to change notification settings - Fork 584
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Considerably high metric cardinality #1479
Comments
Data from prometheus query Just after installation: |
These metrics come from controller-runtime, it's the Kubernetes SDK that we are using to develop the GitOps toolkit controllers. Feel free to create Prometheus rules and drop things that you don't need or open an issue on controller-runtime. |
The API metrics are used here https://github.com/fluxcd/flux2/blob/main/manifests/monitoring/grafana/dashboards/control-plane.json#L332 |
Sorry, but that is only an excuse and not really a fix :) The issue is still present in flux, even if it cause by an upstream library misbehavior. For anyone who finds this issue in the future, here is a relablling that removes all |
We also encountered this issue and had to disable prometheus scraping for flux, as the costs were not justifyable. It has been fixed in the |
We are rolling the update to all Flux controllers, in the latest release some of them are already on controller-runtime v0.10.2. Once all of them will be updated I will close this issue. |
As of flux 0.24.0, all controllers have been update to controller-runtime v0.10 so this issue is finally fixed. Now we need to remove the graph using |
Describe the bug
Basic installation of fluxv2 produces ~6000 metric series. Majority (~5000) of those come from
rest_client_request_latency_seconds_.*
buckets. As far as I can see, only a small subset of data from those metrics is actually used (I found them used only in one panel in "Flux Control Plane" dashboard).Are those used for anything else? If so, maybe there would be a way to reduce their cardinality?
To Reproduce
Steps to reproduce the behaviour:
https://github.com/fluxcd/flux2/releases/download/v0.14.2/install.yaml
to a clusterExpected behavior
Less metrics, but of high quality.
Additional context
Below please provide the output of the following commands:
The text was updated successfully, but these errors were encountered: