Upon append of a new series, should watermarks be written immediately?

[juliusv]

See my new comment on already merged earlier PR: https://github.com/prometheus/prometheus/pull/291/files#r4810106

[juliusv]

Right now we have to wait for the curator to run until we can query a completely new series :-/

[matttproud]

I would suggest checking whether in-memory series for the given fingerprint is empty before deeming the fingerprint "too old". This would be best, as the watermarks ideally should reflect disk state.

[juliusv]

To narrow the error possibility down more: we can query series which a) have been curated before (have a disk watermark), or b) have received a memory sample already during the current Prometheus run. The edge case are timeseries which have not been curated and which haven't been seen yet since the last restart. This can happen if you briefly run a new Prometheus, then restart it. Suddenly you are not able to query the samples from the first run anymore (until you receive more of the same timeseries).

[juliusv]

Done in a1a23fb

We also found out that the curator actually never writes watermarks for series that don't have watermarks already. So did we never actually have watermarks before?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.