add ability to set max resolution

[Andor]

Currently, we have hardcoded limit for 11000 data points per query.
I think we should make it configurable. For some cases, we need to limit it to less than 4096, sometimes we need higher resolution more than 11000.

prometheus/web/api/v1/api.go

Line 202 in 30af475

if end.Sub(start)/step > 11000 {

	// For safety, limit the number of returned points per timeseries.
	// This is sufficient for 60s resolution for a week or 1h resolution for a year.
	if end.Sub(start)/step > 11000 {
		err := errors.New("exceeded maximum resolution of 11,000 points per timeseries. Try decreasing the query resolution (?step=XX)")
		return nil, &apiError{errorBadData, err}
	}

[brian-brazil]

Can you explain in what cases you need more than 11k points, and why you can't handle it in multiple requests?

[olivernadj]

Hi Brian,
We use Prometheus for system, application and business metrics and provide access for system engineers, developers, QA and even non IT guys.
Since many associate can build dashboards in Grafana it follows that some of them are performing slowly and one of the reason is the non optimal resolution.
In the office I did not find any person who has wider screen than 4096 pixel, but I find many dashboards try to represent few thousands data points in a small graph, what causes unintended overload.
To keep Prometheus responsive we decreased max response time, but we would also like to decreased max resolution as I prefer a die quick approach.

[Daemon-Knoppix]

Hi Brian,

I am new to machine learning and monitoring servers and using Prometheus 2.0.0 on ubuntu 16.04 and monitoring stats over a period of time.

My scraping cycle is 1s that means for getting stats overtime for 1 day i need to process 86400 datapoints (24 × 60 × 60) (which is pretty reasonable) i have to call query range api about 8 times as per current limit 11000.

When i start monitoring datapoints for a month (assuming 31 days) value of datapoints jumps to 2678400 datapoints ( (24 × 60 × 60) * 31 ), which as per current limit of 11000 becomes query range api call of about 243 times.

It would be great if this setting could be introduced for user to decide how much datapoints they need at a time.

*Assuming this setting will be added in future, doing this might take time ... Can you please guide me on how and where can i change this on my Ubuntu source code ? As it only shows me binary code which is not human readable.

[brian-brazil]

You probably want the query endpoint there to get raw data, not query range which is evaluating promql.

That sort of massive query is why we have this limit :)

[Daemon-Knoppix]

Thanks for a quick reply!!

True, datapoints for a month will result in a very big data set ... How can i achieve query endpoint for raw data ? Can you share a tutorial or a guide ?

[brian-brazil]

https://prometheus.io/docs/prometheus/latest/querying/api/#instant-queries

[Daemon-Knoppix]

Brian, this query provides data for a single instance in time (current or at a single point), not a solution to what i am looking for, i am evaluating data for a month ... Let me explain -

Lets say i want to check how many times my server was down over a month (i.e up == 0). Now if i use instant queries it will give me record of a single instance in time that means for a month i will end up fetching data from 2678400 api calls. Query range however provides data for 11000 datapoints at once only.

I have build a code in NodeJS (which at my end is very much optimized) that divides 2 dates ranges into intervals as per maximum limit (which in this case is 11000) and make api calls to fetch data.

When i process data for a month i hit api call 243 times (asynchronous parallel calls using async module) which is taking overall time 70 - 80 seconds to fetch data for a month (for a single instance i am monitoring 24*7).

Now if the query was to status code of my application 200, 400 etc.. This single api time increases to 3 to 5 seconds which makes overall process exceed limit.

So i can now see 2 ways to get faster results for getting data for a month -

1. Limit of 11000 datapoints is increased at Prometheus end.
2. I create my own data parser on location Prometheus is storing its datapoints.

Considering option 2 : As i have not looked at source code of Prometheus it will be a little tricky for me to perform this in a short time frame. If i can get a guide on how your query engine works / stores datapoints i might end up creating / enhancing my own logic in it or externally.

Considering option 1 : If increasing datapoint limit at Prometheus end is not that of a big deal in terms of performance, as it's query engine must be very much optimized by you guys ... I would like to know how can i do that in Ubuntu 16.04 setup of Prometheus or change in source code and build a new setup from current updated source.

Please let me know which option would you suggest. Thanks in advance!!!

[gouthamve]

@Daemon-Knoppix Append the time range to the end of the query string. Ex: http://localhost:9090/api/v1/query?query=up[10h]&time=1511347941.586

Also, please ask further queries like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.