Improperly filling in null values

[rrusso1982]

Proposal

When we are querying for a range of data with grafana in which the query time range would end after the last datapoint in prometheus instead of returning null datapoints we instead get a duplicate fo teh last datapoint repeatedly.

Bug Report

We are ingesting metrics from AWS cloudwatch and that data has a delayed availablility in the AWS cloudwatch API. Because of that our exporter emits the timestamp of the cloudwatch datapoint with the export. On average this means we have datapoints that are 5 minutes old as the last point in prometheus. When queried for a range of from 15 minutes ago til now we get all the data points up to the last ingested point and then a repeat of that last point up until now.

What did you expect to see?
I would expect that we should get null data points

What did you see instead? Under which circumstances?
We get duplicate data points of the last available point

Environment
Cloudwatch -> Prometheus -> Grafana

• Prometheus version:

2.3.2

[codesome]

I think this is by design. What you are looking for might be the --query.lookback-delta flag, which defaults to 5m. So when you query, the latest sample within the last 5m is returned. If there is no sample within that range, the series would be marked stale.

[simonpasquier]

What @codesome says. This is documented here.

I'm closing it for now. If you have further questions, please use our user mailing list, which you can also search.