increase function returns null for some data points, resulting in gaps

[Vlad777]

I have complete data for "mymetric" in prometheus for every second, with no nulls. I use increase function on a counter metric over a period of 1m to see the increase for every minute: increase(mymetric[1m]).

I expect a value of the increase from first to last data point of each minute. When there was no increase, I expect to see a 0.

Instead, I often see unexplainable gaps (null values) in the data; even when trying with other functions like delta and rate.

Prometheus version: 1.5.2
BuildDate: 20170210-14:41:22
GoVersion: go1.7.5

[juliusv]

for every second

Is your scrape interval really 1s? The only way I could explain missing values is if rate() (or increase() or delta()) does not manage to find at least two samples under the 1-minute window you supply.

[Vlad777]

I do see data being returned for every second. I have exported into excel and verified this. Here is a screenshot of the raw data and the result of increase function that shows the null gaps.

[Vlad777]

Here is how it looks with idelta function which actually makes sense and has no gaps.

[juliusv]

Ok, if idelta() with the same range window returns values in places where rate() doesn't, I'm a bit stumped. If you look at the raw returned AJAX JSON data, do you see explicit NaN / +Inf / -Inf datapoints for the rate() in the gappy places or just no samples at all?

[juliusv]

If you look at the raw returned AJAX JSON data

And with that I mean: open browser's inspector, network tab, refresh graph, look at AJAX response payload.

[Vlad777]

The timestamps that match the gaps are simply missing from the result. I only see valid numbers and 0 values, without explicit NaN/ +Inf / -Inf values.

[brian-brazil]

Can you please post the exact expression you're using?

[Vlad777]

These are the 2 queries that are plotted as I already mentioned:

mymetric{stack=~"mystack.+"}
increase(mymetric{stack=~"mystack.+"}[1m])

[brian-brazil]

We're going to need the raw data here. Grafana cannot show the raw data, use the console view of the expression browser. Choose one relevant time series.

[brian-brazil]

That's not the raw data, as it seems very unlikely to me that your data would be exactly on the second. Please use the expression browser console view to get it.

I also note you're using the pushgateway what appears to be a non-batch job, this is not recommended. https://prometheus.io/docs/practices/pushing/

[juliusv]

To clarify more, you will need to run an instant (console / table) query instead of a graph query in the expression browser, and then maybe query for one of the buggy series:

foo[1h]

This gives you from now to 1h back all the raw samples of that series.

[Vlad777]

This is the query being run:
http://hostname/api/datasources/proxy/2/api/v1/query_range?query=mymetric%7Bstack%3D~%22mystack.%2B%22%7D&start=1493392501&end=1493394401&step=1

[juliusv]

Right, that's a range query. What we'd need to see the truly raw data is an instant query that includes a time range window. Something like: http://demo.robustperception.io:9090/api/v1/query?query=alertmanager_alerts_received_total%7Bstatus%3D%22firing%22%7D%5B4h%5D&time=1493402433.899&_=1493402397084

[Vlad777]

Thank you for clarifying. Looks like the instant query shows that there are periods that are larger than 60 seconds between timestamps, which as you mentioned would be the cause of the gaps in the delta calculations as well.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.