If throttled, don't `rate`, etc. with 0 values

[Dominik-K]

We've run into the "throttling mode" ° today. It seems that Prometheus (1.6.3) uses zero values for the throttled time and calculates wrong rates resulting in false alerts afterwards.

A threshold-underrun alert was thrown 60 minutes later because we used a rate(...[60m]) function:

• Throttling started at 16:27 and left "rushed mode" short before 16:42.
• Both graphs are based on the same sum(rate(METRIC)) with different rate intervals.

What did you expect to see?

• A gap between 16:27 and 16:42 as Prometheus didn't scrape any values at this time.
• A near-to-flat, small changing rate(...[60m]) graph (as seen before 16:27 and after 17:45).
  => No false alerts like threshold was underrun.

I propose that Prometheus skips the throttled timeframe. I.e. uses the last value before throttling for rates calculated after it left "rushed mode", e.g. has real data values again.

° "Storage needs throttling. Scrapes and rule evaluations will be skipped." (urgencyScore=1)

[brian-brazil]

Nothing is ingested when throttling.

You're not showing us everything here. Is there a single time series which shows the supposed problem?

[Dominik-K]

The graphs shown above (and below) are based on the same counter timeseries. Only the rate interval is different. Here's also the raw counter value shown in the same graph (times are UTC+1 with summer time => +2h offset to UTC+0):

The raw counter value in Prometheus (times are UTC+0):

While throttling at the first 5 minutes, the counter value is set with the same value as the last scrape. Then there are NaN's until throttling is over. That's a second problem, though.

[brian-brazil]

Can you share the raw data for that counter?

[Dominik-K]

Sure. Here are the raw values, retrieved with the query_range API call.

[brian-brazil]

Can you get the values using the query api please.

[Dominik-K]

Where's the difference? I just checked the first and last value with the query call and they are the same. E.g. the last value in the timeframe:

{
    "status": "success",
    "data": {
        "resultType": "vector",
        "result": [
            {
                "metric": {
                    "__name__": "REDACTED",
                    "instance": "REDACTED",
                    "job": "REDACTED"
                },
                "value": [
                    1496332800,
                    "266768234"
                ]
            }
        ]
    }
}

[brian-brazil]

query_range does not return the raw data.

[brian-brazil]

I'm presuming this is a misunderstanding of how rate() works. If you've evidence that rate() is broken, please share the raw data.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.