Some rate calculations on rc1 are incorrect

[smarterclayton]

In 2.0.0-rc1 some queries for rate over counters seem to be wildly overstated / incorrectly calculated. This is very large kubernetes cluster that is receiving ~0.1 requests per second on average for a specific request error code, but the rate calculation is reporting 28 req/s.

Here's the samples from the three masters for a particular metric apiserver_request_count sampled from the server directly for one server:

apiserver_request_count{client="openshift/v1.7.0+80709908fd (linux/amd64) kubernetes/b0608fa",code="404",contentType="application/vnd.kubernetes.protobuf",resource="secrets",scope="resource",subresource="",verb="GET"} 7584

... wait 5-10s

apiserver_request_count{client="openshift/v1.7.0+80709908fd (linux/amd64) kubernetes/b0608fa",code="404",contentType="application/vnd.kubernetes.protobuf",resource="secrets",scope="resource",subresource="",verb="GET"} 7584

... wait 5-10s

apiserver_request_count{client="openshift/v1.7.0+80709908fd (linux/amd64) kubernetes/b0608fa",code="404",contentType="application/vnd.kubernetes.protobuf",resource="secrets",scope="resource",subresource="",verb="GET"} 7587

Prometheus console reports the correct instantaneous value:

Query:
apiserver_request_count{client="openshift/v1.7.0+80709908fd (linux/amd64) kubernetes/b0608fa",code="404",contentType="application/vnd.kubernetes.protobuf",resource="secrets",scope="resource",subresource="",verb="GET"}

Result:
{client="openshift/v1.7.0+80709908fd (linux/amd64) kubernetes/b0608fa",code="404",contentType="application/vnd.kubernetes.protobuf",instance="172.31.21.249:443",job="kubernetes-apiservers",resource="secrets",scope="resource",subresource="",verb="GET"} | 7587

Over the last six hours the total has only increased by about 3k which should be a rate of about 0.15 sec.

But the query for rate is reporting a 5m rate of 28/sec which is impossible:

Query:
rate(apiserver_request_count{client="openshift/v1.7.0+80709908fd (linux/amd64) kubernetes/b0608fa",code="404",contentType="application/vnd.kubernetes.protobuf",instance="172.31.21.249:443",job="kubernetes-apiservers",resource="secrets",scope="resource",subresource="",verb="GET"}[5m])

Result:
{client="openshift/v1.7.0+80709908fd (linux/amd64) kubernetes/b0608fa",code="404",contentType="application/vnd.kubernetes.protobuf",instance="172.31.21.249:443",job="kubernetes-apiservers",resource="secrets",scope="resource",subresource="",verb="GET"} | 27.712352499999998

[smarterclayton]

The prometheus instance has been running for about 20 hours since it was last cleared and reset due to #3283 and #3316

[smarterclayton]

Other rates on other series are also broken by two or three orders of magnitude

[fabxc]

Thanks, does this sound like #3337?

[smarterclayton]

Yes, will check the samples reported in the morning

[smarterclayton]

Looks like this was fixed with rc2:

[smarterclayton]

There's a tiny tail of correct rate on the right edge.

[fabxc]

Awesome, thanks for the quick response!

[paultiplady]

@fabxc any idea when this regression was introduced? I seem to be hitting this on 1.7.0, is that possible? Thanks!

[fabxc]

It was caused purely by an issue in the new storage issue which has since then be fixed. If you seeing anything like this in 1.x code, please open a separate issue since it's most definitely another problem.

…

On Thu, Dec 14, 2017 at 1:21 AM Paul Tiplady ***@***.***> wrote: @fabxc <https://github.com/fabxc> any idea when this regression was introduced? I seem to be hitting this on 1.7.0, is that possible? Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3345 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEuA8l99dFYpSk3k_WeQLuFQbwhyYAWUks5tAGn6gaJpZM4QEqeg> .

[paultiplady]

Thanks for the quick response -- I'll do some more digging on my side and try upgrading to see if that clears things.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.