delta, rate, irate over-report by up to 100%

[vgough]

What did you do?
delta(x[2m]) when collecting at 1m intervals shows 2x the expected values. The over-reporting appears to be based on a ratio of aggregation window and collection windows. If you aggregate over 5m window, then over-reporting drops to 25%. At 10m aggregation window, it drops to 11%.

Can also be seen using this test:

--- a/promql/testdata/functions.test
+++ b/promql/testdata/functions.test
@@ -117,6 +117,15 @@ eval instant at 20m delta(http_requests[20m])
 
 clear
 
+# Tests for delta().
+load 1m
+       http_requests{path="/foo"}      0 0 0 0 1 1 1 1
+
+eval instant at 290s delta(http_requests[2m])
+       {path="/foo"} 1
+
+clear
+
 # Tests for idelta().
 load 5m
        http_requests{path="/foo"}      0 50 100 150

--- FAIL: TestEvaluations (2.13s)
        promql_test.go:33: error running test testdata/functions.test: error in eval delta(http_requests[2m]): expected 1 for {path="/foo"} but got 2
FAIL

Seems like a problem with how sample bounds impact the delta computation: https://github.com/prometheus/prometheus/blob/master/promql/functions.go#L117

Environment
Linux, multiple versions, distributions, kernels.

• Prometheus version:
  production versions of prometheus, and github master.

[brian-brazil]

This behaviour is correct, see https://www.youtube.com/watch?v=67Ulrq6DxwA

You should also not use delta, you want rate, irate or increase here.

[vgough]

If you look at the test diff I sent, the difference from the test directly above it is that I'm asking for a delta on a range that doesn't land exactly upon a sample time. Almost all the other tests run exactly on a sample point, which doesn't exercise the extrapolation code. In real world usage, I've never seen promql returning results synchronized with samples.

Also, as the bug name states, rate and irate similarly report incorrect results. Here's what this looks like from a user's point of view. We have a gauge holding the number of open file descriptors. It isn't easy to see in this snapshot, but the value is almost always 123, sometimes down to 122, sometimes up to 124.

Now take a delta:

The delta always shows increments and decrements by factors of 2. That never occurs in this graph.

irate also shows increments as a factor of 2. These graphs are taken from prometheus 1.6.1's graph interface.

The video you linked is 40minutes long. If you'd like to reference a particular part, I'm happy to take a look. However the results we're seeing are creating misleading graphs, and clearly impossible graphs are how I discovered the issue. In the short term, we've started using correction factors to correct the graphs, eg divide by 2 when using 2m aggregation, divide by 1.25 when using 5m aggregation, etc.

[brian-brazil]

The entire talk is relevant, as it is far more likely that you are misunderstanding the behaviour rather than there being a bug.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.