Low-resolution querying may drop long time ranges

[fabxc]

When querying Prometheus with "low" resolution (default is often low enough), minute long time ranges are often not shown. Those seem to correlate exactly with blocks, but not confirmed yet.

Setting an explicit low query resolution makes the expected data reappear in the results.

[fabxc]

I'm seeing this pretty consistently while @gouthamve cannot reproduce this. Would be good to know whether other's can. @mwitkow @grobie @beorn7

[mwitkow]

@fabxc I'm not sure what you mean by low resolution? A narrow time window?
Can you give an example of a /query param to try reproduce?

[fabxc]

Just range queries with a coarse-ish step size.

[mwitkow]

Can confirm. Querying for 5m intervals work, but anything like 1m or 30s rates show holes in data.

Take a look at this. There is no gap gap around 11:00 when using 5m rate

vs the massive gap in served responses (the cursor "skips oveer") around 11:0 when using 30s

[fabxc]

Thanks for confirming – now to find out where it all went wrong.

[gouthamve]

I can reproduce this locally now, some early findings:

1. Low resolution is not the cause of it: Low res vs High res
2. The same resolution over different time ranges can cause it: shorter range vs longer range

And the worst part is, in 2) above, the break happened at 1800, and the block spans from 1315 to 2000, so this is definitely not due to block boundaries. Digging further.

[gouthamve]

This is caused because of this commit by me. When we cut a chunk, we are not setting a mint and it is defaulting to 0.

Later we are Seeking to the right chunk using the chunk's MinTime which makes us seek to wrong chunks and skip some chunks. Fixing the mintime when cutting a chunk should fix this IMO.

[fabxc]

Chunk's min timestamp is fixed as part of prometheus/tsdb#94. Will check whether I can still produce the issue.

[mwitkow]

Can't reproduce it from our side with today's dev-2.0 cut so 🍾

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.