Monitoring Dashboards: Use /query_range to fetch label values #4099
Conversation
Use `/query_range` instead of `/query` to fetch the available label values so that all values during the time range are returned, not just the values available right now. Using `/query_range` is not perfect because it is not 100% guaranteed to return all values. Specifically, short lived series could be missed by `/query_range`'s data sampling. Although this should be very unlikely in practice. Using `/series` would be guaranteed to return all series, but it is not yet available through our API proxy. Using `/series` should also be more efficient. We also can't use `/label/<LABEL>/values` because it doesn't take a label selector.
openshift-ci-robot
added
size/S
|
As discussed yesterday OOB this is to my knowledge the current low-hanging fruit workaround. /cc @openshift/openshift-team-monitoring |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: kyoto, s-urbaniak The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
cc @simonpasquier how difficult would it be to add the series endpoint filtering to the prom label proxy? The way I see it is just doing the request and filtering for the configured label, seems like a pretty quick change no? |
|
fwiw created https://issues.redhat.com/browse/MON-960 so we can track this internally. |
|
@brancz yes it should be easy to do. This is similar to what we're doing for the alerts and rules endpoints. |
|
If you think you can get it done in a reasonably time boxed manner, I'd be more than happy to review that. Query range is somewhat problematic performance wise. |
Use
/query_rangeinstead of/queryto fetch the available labelvalues so that all values during the time range are returned, not just
the values available right now.
Using
/query_rangeis not perfect because it is not 100% guaranteed toreturn all values. Specifically, short lived series could be missed by
/query_range's data sampling. Although this should be very unlikely inpractice.
Using
/serieswould be guaranteed to return all series, but it is notyet available through our API proxy. Using
/seriesshould also be moreefficient.
We also can't use
/label/<LABEL>/valuesbecause it doesn't take alabel selector.
In future, we should aim to make the Prometheus
/seriesendpoint available through our API proxy.FYI @s-urbaniak