/graph needs a way to select the "top 5" or similar

[leonerd]

I have a timeseries with about 40 or so values for a label. I want to show a graph of the top 5 of them. If I simply plot

topk(5, ...)

then this picks the top five for each time in the graph, resulting in at most 5 non-zero lines per vertical slice of time, but there's still around 30 or so plots overall.

Instead, the selection of the "top five" needs to be done after the entire set of values is collected ready for display by the graph generator. It would have to be an argument to the graph generator itself.

---

Additionally, it would be really nice if the remaining buckets could be summed into an "everything else" plot, so that I could draw a stacked area graph showing the top 5 in the time period, and all the rest as one final area, so that the total is still representative.

[discordianfish]

Hi,

just ran into the same issue. I guess as long as the topk timeseries stay the same across your graph, you're fine but if they are not you end up with tons of graphs and missing datapoints (if I understand the issue correctly)

/cc @juliusv @beorn7

[juliusv]

@discordianfish topk is working as intended (different time series are the topk at different times of the graph), but this sounds like a request for a new feature: show the topk ovrerall within a given time range (probably determined by the total area under each time series over the span of the graph time range). This will be much harder to implement, as it doesn't really fit nicely into how we analyze and execute queries. Haven't really put too much thought into it yet though.

[discordianfish]

@juliusv Yes I understand that and topk makes sense as it. Still, having something similar to topk which returns the timeseries with the highest average value over the queried range would be super useful. I think it's a common thing to have dashboards with the top n of some things and with topk you get basically just garbage if the metrics are spiky.

PS: This btw also happens if all timeseries are 0, in that case topk looks pretty random. Maybe we can improve that specific case easily? Guess we could look at the labels or something if the value is the same.

[thefallentree]

we can let user specify another time-range parameter (for example use last 5m to determine topK and show a 1day graph) / or even queries, use to determine the resulting timeseries to show on the graph.

[beorn7]

You could do things like

some_metric and topk(5, avg_over_time(some_metric[5m])

That would at each point display the top 5 metrics based on the average over the last 5m. That helps in some cases, but it's still not "fixed for exactly the displayed period, take the top5 metrics over that period". For that, we needed a way to select a fixed (not relative) time range in the expression language).

[brian-brazil]

I think this is something we should handle at the browser level rather than at the promql level as it doesn't quite fit the computational model.

[leonerd]

Being in the browser would be fine. The client-side rendering JS is probably in a better position to decide which are the "best" k series to plot anyway.

[or4cle]

+1 This would be a very useful to have.

[brian-brazil]

I don't think we're ever going to do better here, the request is not compatible with the Prometheus evaluation model.

[discordianfish]

@brian-brazil What about supporting this client-side? Should we open a new issue for that?

[r4j4h]

Just wanted to mention some workarounds to achieve parts of what the OP wants that have gone unmentioned here:

In Prometheus, the Table view presents only the "current" topk which should only be the k you entered. You can use offset to explore temporally. This should never show more records than the given k.

If you are only wanting to see the history of the current topk, use the table view to grab the currents and then query not for the topk but for just those k records.

IOTW, let's say you had topk(2, my_metric) and it gave back server_a and server_c.
Then I would do a 2nd query for my_metric{server=~"server_a|server_c"} and you should get just the data for your two entries. Hopefully that makes enough sense to extrapolate to your purposes.

Note also you could handle the "rest" in a similar way via negated regexp !~, logical binary like unless, or fancy subtraction or something. Could use sum to unify and/or label_replace to give "Others" name.

Since it hasn't been mentioned, even though it is an additional component, Grafana can do this quite easily as well. It can even happen accidentally on Panel types that prefer to only show "one" value if the dashboard designer isn't careful to use Prometheus' averaging functions and Grafana's "Current" value instead.

To be clear I am not recommended re-opening this issue, just wanting to provide some other options for people.

[brian-brazil]

grafana/grafana#7664 will allow for this in Grafana

[fho]

@brian-brazil how can I create graph showing only the top X by label with using the $__interval option from grafana/grafana#7664?

In my case I have a metric called process_resident_memory_bytes, it has a service_name label per application.
I want to display the 10 services that had the highest process_resident_memory_bytes value on an arbitrary point in the time in the chosen time interval.

If I make a query like
topk(3, max(max_over_time(process_resident_memory_bytes[$__interval])) by (service_name)).
I still get more than 3 values as result because the query returns for every time step in the interval a result.

[brian-brazil]

This is possible in Grafana 5.3.0: https://www.robustperception.io/graph-top-n-time-series-in-grafana