Graph is flickering

[ricardoatsouza]

What did you do?

I'm using Prometheus to gather metrics from a few node js services (Node JS library being used: https://github.com/siimon/prom-client). The data is being collected, so far so good.

This metric in particular is an integer (0, 1, 2, 3, ...), and this is how I see it in Prometheus graph.

But, when I update the graph (same metric, of course), it changes to:

This is a very simple graph, showing four lines of integer number (one line per service) and, every time there is an update in the graph, those values keep rounding -1 down.

This behaviour seems to be happening with floating number as well, as I see this in CPU and memory graphs too. Here is the CPU usage (delta(process_cpu_seconds_total{service_name="myservice"}[5m])) of one of the services:

And, same graph, just refreshed:

I would guess that Prometheus is trying to round the data, but I cannot tell for sure.

Thanks

What did you expect to see?

The graphs to be concise.

What did you see instead? Under which circumstances?

Graphs are flickering.

Environment

• System information:

Output of uname -srm: Darwin 17.4.0 x86_64

• Prometheus version:

  build user:       root@149e5b3f0829
  build date:       20180314-14:15:45
  go version:       go1.10

• Prometheus configuration file:

global:
  scrape_interval:     5s
  evaluation_interval: 5s

  external_labels:
      monitor: 'my-monitor'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'services'
    static_configs:
      - targets: [
        's1:3000',
        's2:3000',
        's3:3000',
        's4:3000',
        's5:3000'
      ]

[brian-brazil]

This is expected, when you refresh you request data at slightly different times so the result of the math changes. You should also not be using delta with a counter, you want increase instead.

[ricardoatsouza]

Care to explain why this is expected? If the data point was collected at a timestamp t, and the x-axis of the graph includes t, why would the the data at t flicker?

And why you assumed I'm using a counter? It's a gauge (that particular number also decreases).

[brian-brazil]

You didn't request the raw data, you requested query evaluation so it's evaluating queries at each step at returning (roughly) the point before that time. The two queries you are using have different values for t.

process_cpu_seconds_total is a counter.

[ricardoatsouza]

Hmm... I get it. Probably the I'm misusing the delta function in there.

But, should I expect to see it also when I request only the metric itself?
In my first example, I'm just selecting it from the dropdown menu and clicking in execute.

[brian-brazil]

The graph tab offers only promql execution, so that's an instant vector selector being evaluated. If you wanted raw data you'd need to query metricname[5m] on the console tab.

[ricardoatsouza]

Indeed. Wasn't aware of the differences in between selecting via the dropdown and by typing it in the console tab. I probably missed that in the documentation.

But still, even though if this is normal/ intended behaviour, would it be possible to make the graph tool reflect exactly the output of the console? I'm just asking because it's weird to see the graph flickering, specially when someone (like me) doesn't know that this can happen.

[brian-brazil]

The graph and the console do have the exact same output for a given timestamp, it's all the same code. The graph tab does not support raw data though.

[ricardoatsouza]

The graph and the console do have the exact same output for a given timestamp, it's all the same code.

Yes, I would expect that. Makes total sense and sounds logical.

The graph tab does not support raw data though.

I'm aware that plotting data in graphs might involve issues like this. My point is, does it makes sense to make it support the raw output at some point in the future? It is just a suggestion, and, since there are tools like Grafana, it might not make sense at all to make it support the raw data.

[brian-brazil]

The raw data is only used generally when doing deep debugging, even Grafana doesn't support graphing it.

[ricardoatsouza]

Then my suggestion doesn't make sense.
Thanks for clarifying the issue.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.