New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

/graph needs a way to select the "top 5" or similar #586

Closed
leonerd opened this Issue Mar 9, 2015 · 14 comments

Comments

Projects
None yet
10 participants
@leonerd

leonerd commented Mar 9, 2015

I have a timeseries with about 40 or so values for a label. I want to show a graph of the top 5 of them. If I simply plot

topk(5, ...)

then this picks the top five for each time in the graph, resulting in at most 5 non-zero lines per vertical slice of time, but there's still around 30 or so plots overall.

Instead, the selection of the "top five" needs to be done after the entire set of values is collected ready for display by the graph generator. It would have to be an argument to the graph generator itself.


Additionally, it would be really nice if the remaining buckets could be summed into an "everything else" plot, so that I could draw a stacked area graph showing the top 5 in the time period, and all the rest as one final area, so that the total is still representative.

@leonerd leonerd changed the title from topk() does not play nicely on /graph to /graph needs a way to select the "top 5" or similar Mar 9, 2015

@discordianfish

This comment has been minimized.

Member

discordianfish commented May 12, 2015

Hi,

just ran into the same issue. I guess as long as the topk timeseries stay the same across your graph, you're fine but if they are not you end up with tons of graphs and missing datapoints (if I understand the issue correctly)

/cc @juliusv @beorn7

@juliusv

This comment has been minimized.

Member

juliusv commented May 12, 2015

@discordianfish topk is working as intended (different time series are the topk at different times of the graph), but this sounds like a request for a new feature: show the topk ovrerall within a given time range (probably determined by the total area under each time series over the span of the graph time range). This will be much harder to implement, as it doesn't really fit nicely into how we analyze and execute queries. Haven't really put too much thought into it yet though.

@discordianfish

This comment has been minimized.

Member

discordianfish commented May 13, 2015

@juliusv Yes I understand that and topk makes sense as it. Still, having something similar to topk which returns the timeseries with the highest average value over the queried range would be super useful. I think it's a common thing to have dashboards with the top n of some things and with topk you get basically just garbage if the metrics are spiky.

PS: This btw also happens if all timeseries are 0, in that case topk looks pretty random. Maybe we can improve that specific case easily? Guess we could look at the labels or something if the value is the same.

@thefallentree

This comment has been minimized.

thefallentree commented May 16, 2015

we can let user specify another time-range parameter (for example use last 5m to determine topK and show a 1day graph) / or even queries, use to determine the resulting timeseries to show on the graph.

@beorn7

This comment has been minimized.

Member

beorn7 commented Jul 15, 2015

You could do things like

some_metric and topk(5, avg_over_time(some_metric[5m])

That would at each point display the top 5 metrics based on the average over the last 5m. That helps in some cases, but it's still not "fixed for exactly the displayed period, take the top5 metrics over that period". For that, we needed a way to select a fixed (not relative) time range in the expression language).

@brian-brazil

This comment has been minimized.

Member

brian-brazil commented Jul 15, 2015

I think this is something we should handle at the browser level rather than at the promql level as it doesn't quite fit the computational model.

@leonerd

This comment has been minimized.

leonerd commented Aug 4, 2015

Being in the browser would be fine. The client-side rendering JS is probably in a better position to decide which are the "best" k series to plot anyway.

@or4cle

This comment has been minimized.

or4cle commented Mar 17, 2016

+1 This would be a very useful to have.

@brian-brazil

This comment has been minimized.

Member

brian-brazil commented Feb 13, 2017

I don't think we're ever going to do better here, the request is not compatible with the Prometheus evaluation model.

@discordianfish

This comment has been minimized.

Member

discordianfish commented Feb 13, 2017

@brian-brazil What about supporting this client-side? Should we open a new issue for that?

@r4j4h

This comment has been minimized.

r4j4h commented May 16, 2017

Just wanted to mention some workarounds to achieve parts of what the OP wants that have gone unmentioned here:

In Prometheus, the Table view presents only the "current" topk which should only be the k you entered. You can use offset to explore temporally. This should never show more records than the given k.

If you are only wanting to see the history of the current topk, use the table view to grab the currents and then query not for the topk but for just those k records.

IOTW, let's say you had topk(2, my_metric) and it gave back server_a and server_c.
Then I would do a 2nd query for my_metric{server=~"server_a|server_c"} and you should get just the data for your two entries. Hopefully that makes enough sense to extrapolate to your purposes.

Note also you could handle the "rest" in a similar way via negated regexp !~, logical binary like unless, or fancy subtraction or something. Could use sum to unify and/or label_replace to give "Others" name.

Since it hasn't been mentioned, even though it is an additional component, Grafana can do this quite easily as well. It can even happen accidentally on Panel types that prefer to only show "one" value if the dashboard designer isn't careful to use Prometheus' averaging functions and Grafana's "Current" value instead.

To be clear I am not recommended re-opening this issue, just wanting to provide some other options for people.

@brian-brazil

This comment has been minimized.

Member

brian-brazil commented May 17, 2017

grafana/grafana#7664 will allow for this in Grafana

@fho

This comment has been minimized.

fho commented Sep 28, 2017

@brian-brazil how can I create graph showing only the top X by label with using the $__interval option from grafana/grafana#7664?

In my case I have a metric called process_resident_memory_bytes, it has a service_name label per application.
I want to display the 10 services that had the highest process_resident_memory_bytes value on an arbitrary point in the time in the chosen time interval.

If I make a query like
topk(3, max(max_over_time(process_resident_memory_bytes[$__interval])) by (service_name)).
I still get more than 3 values as result because the query returns for every time step in the interval a result.

@discordianfish discordianfish referenced this issue Oct 2, 2017

Closed

Sorted stacked graph #5744

1 of 3 tasks complete

simonpasquier pushed a commit to simonpasquier/prometheus that referenced this issue Oct 12, 2017

Merge pull request prometheus#586 from prometheus/beorn7/doc-improve
Clarify that rule files are configured via globs
@brian-brazil

This comment has been minimized.

Member

brian-brazil commented Oct 15, 2018

This is possible in Grafana 5.3.0: https://www.robustperception.io/graph-top-n-time-series-in-grafana

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment