Record rules results shifted in time

[alex-k]

Bug Report

What did you do?

I have two recording rules

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s


record: sli:metric:good_events 
expr: sum   by(product, journey, step) (splunk_journey{data="good_events"}) 
labels:   metric: splunk_journey

record: sli:metric:valid_events 
expr: sum   by(product, journey, step) (splunk_journey{data="valid_events"}) 
labels:   metric: splunk_journey

What did you expect to see?

I expect to see in result same values as in target metrics grouped by labels

What did you see instead? Under which circumstances?

Sometimes I see values shifted in time by 15sec

Expected:

Got:

Environment

Not applicable

• System information:

Not applicable

• Prometheus version:

Version 2.6.1
Revision b639fe1

• Alertmanager version:

Not applicable

• Prometheus configuration file:

• Alertmanager configuration file:

• Logs:

[roidelapluie]

Can you share your recording rules file?

[alex-k]

Sure,

https://gist.github.com/alex-k/d8fbcc8709a09833ee9d83ac97389de2

[kminehart]

This is similar to:

#4860

I believe we were experiencing the same sort of time drift problem, but we have way too many recording rules, and a lot of them are fairly complex (most of them came with the prometheus operator), so we might have been experiencing an even greater amount of it.