Limit number of metrics displayed in prometheus UI

[discordianfish]

IMO it's too easy to crash the browser by entering a expression returning too many elements. It should be straight forward to only display the first X elements a query returns and print a warning.
X should be really high so the limit only kicks in in cases where displaying the data would takes several gigs of memory.

[gouthamve]

I think this is related to #2253

Fixing one will fix another (by using a parameter)

[brian-brazil]

That's not related.

[fabxc]

Yes, #2253 talks about the horizontal axis, i.e. how many datapoints are evaluated for any particular expression. That affects any number of time series this expression may select. The problem described here talks about the vertical axis of how many result time series there can be. Probably both should be configurable by the user to be honest. There are many ways how a query can be made expensive enough to hog a Prometheus. Having a hardcoded limit on a single one of them doesn't seem particularly useful.

…

On Tue, Feb 28, 2017 at 9:46 AM Brian Brazil ***@***.***> wrote: That's not related. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2119 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEuA8tHkybl2Fb42u5KhZO3wUp9eVUfTks5rg97IgaJpZM4KgKGd> .

[brian-brazil]

I ran into a case a while back where labels were being put in metric names, and this was taking out my browser (plus chewing through CPU for prometheus). I think a hardcoded limit on the order of 10-100k should do the trick.

[PrincipalsOffice]

Is this issue still up for grabs? I am interested. And can I know the URL for this issue?

[brian-brazil]

Noone is working on this currently. This is the the v1 label API.

[PrincipalsOffice]

Should the limit be implemented on the v1 instead of v2 api then?

[biglovisa]

Even if the results aren't crashing your browser, the way the timeseries are rendered today (both on the Graph and Console view) isn't very easy to read or skim. In addition to limiting the results returned, perhaps better filtering would be useful here as well? For example, you could only display timeseries if certain labels are present, or even specific label-value pairs.

@PrincipalsOffice are you still working on this? If not, I have time to work on this.

[brian-brazil]

Such features are more for Grafana. This issue is about avoiding taking out browsers.

[frittentheke]

This is still very much an issue with Prometheus 2.2.1. Even on a beefy machine with a current browser the UI fetches such large results when doing a character by character typeahead it sometimes even crashes.

Maybe a quick and dirty solution (before applying a limit on the number of results via the API) is to only start the type-ahead after like 3 or 5 characters or to allow disabling it all-together and only start the search after I press enter?

[arnisoph]

This is blocking us. Is there any other method to get a list of (all) available metrics?

[brian-brazil]

This is the only method to get all the metrics, and will remain so. As far as we're aware this only affects browsers when using this UI, not other uses.

[krasi-georgiev]

how difficult would be to have something like Facebook style , display first 1000 and scrolling down reveals more and more lines.
Anyone here familiar with such implementation?

[brian-brazil]

Pretty hard, that requires either state on the backend or pagination - both have performance/complexity implications. In practice if you have more than 100k metrics it's not practical to scroll through them anyway, even 1k is a lot for a human to deal with.

[pramodhkp]

Hi, I'd like to pick this up and give it a try. Any pointers or suggestions as to things to keep in mind and where to start?

Thanks!

[potomak]

In #4765 I limited the number of items in the allMetrics array to be less than 10k. If the number of items gotten from the HTTP response body is more than the 10k threshold a warning message is prepended to the first graph wrapper element.

I'm not sure about this implementation of the warning message thought. I was going to add an alert-warning element right after the alert-error element in the graph wrapper template, but then I decided to copy the way a warning message is displayed in the case there's a time difference between the server and the browser that's greater than 30 secs:

prometheus/web/ui/static/js/graph/index.js

Lines 218 to 222 in 918f841

	$("#graph_wrapper0").prepend(
	"<div class=\"alert alert-warning\"><strong>Warning!</strong> Detected " +
	diff.toFixed(2) +
	" seconds time difference between your browser and the server. Prometheus relies on accurate time and time drift might cause unexpected query results.</div>"
	);