cpu usage sometimes gives negative values

[ghost]

Hello,
I want to run a script periodically which will give me the cpu usage. I am using the following query on influxDb which is a sink for my heapster:

select derivative(value) from "cpu/usage_ns_cumulative" where pod_namespace='scaledemo' AND labels=~/name:locust,role:worker/ AND time > now() - 300s group by pod_namespace, labels order desc"

Result of Query:
[
{
"columns": [
"time",
"derivative",
"pod_namespace",
"labels"
],
"name": "cpu/usage_ns_cumulative",
"points": [
[
0,
-4413325.490494297,
"scaledemo",
"name:locust,role:worker"
]
]
}
]

This gives me sometimes negative values as seen above.

Kindly suggest.

Regards,
Shilpa

[vishh]

I have seen this issue with InfluxDB v0.8.x. One option is to setup continuous queries in InfluxDB to sample the cpu_usage periodically into a new series, and avoid derivatives(..).

Another option is to try InfluxDB v0.9. The configs on Heapster HEAD support InfluxDB v0.9. I have not seen issues with derivative with this version.

cc @piosz who is working on adding instantaneous cpu metrics.

[alexmavr]

As an additional note, this behavior also occurs if the container is restarted between measurements of cpu/usage_ns_cumulative.

[piosz]

@Biskup noticed similar issue in Grafana dashboard. I'll try to fix #568 soon.

[piosz]

There was an int overflow in cadvisor kubernetes/kubernetes#27194. This is fixed now