Make rule evaluation and federation consistent in time

[beorn7]

Currently, if samples get ingested during a rule is evaluated or a federation request is processed, only a subset of samples with the same timestamp may be taken into account during the evaluation.

Making ingestion of all the samples from a single scrape atomic would solve the problem, but the problem might be solved in an easier way by "simply" ignoring samples that came in after a certain watermark.

For reference: discussion started in prometheus/docs#514 and already referenced #398 as a possible part of the solution.

[beorn7]

Also #1887 for the reported incident that caused the discussion in the first place.

[lmb]

I reported #2598, which was closed as a duplicate of this issue. I think this issue should mention that this behaviour breaks aggregating histograms using histogram_quantile in the currently released version. That should prevent more dupes.

[gouthamve]

AFAICS, this is fixed 2.0. Can this be closed @fabxc @brian-brazil

[brian-brazil]

This is not fixed in 2.0 (yet).

[brian-brazil]

I've got a basic version of this working over in the tsdb repo, it needs cleanup and performance work.

[krasi-georgiev]

@beorn7 , @brian-brazil , @gouthamve is this issue still unsolved?

[brian-brazil]

Yes, there's an old PR over in TSDB for this. I believe @gouthamve was planning on working on it soon.

[krasi-georgiev]

aah that must be the isolation PR prometheus/tsdb#306