Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prometheus query time alignment cause bar width to fluctuate #12379

Closed
roidelapluie opened this Issue Jun 22, 2018 · 8 comments

Comments

Projects
None yet
7 participants
@roidelapluie
Copy link
Contributor

roidelapluie commented Jun 22, 2018

bug

to reproduce: create a bar graph panel

zoom it zoom out move forward ... after a few attemps the problem shows.

in real condition it shows up randomly.

@dswarbrick

This comment has been minimized.

Copy link

dswarbrick commented Jun 29, 2018

I just noticed this too, but strangely it only affects data from one particular host so far. I am using node_exporter 0.16.0, and am graphing CPU times by mode (e.g. user, system, iowait etc). I only notice this weird effect for a host that has 64 cores, but I really doubt it's related to that.

I can reproduce the issue by simply clicking on the graph refresh button once per second. The bars get thinner with each refresh, then at the next interval alignment, they are back to their regular width.

@torkelo

This comment has been minimized.

Copy link
Member

torkelo commented Jul 2, 2018

Can you show the query used to replicate this?

@dswarbrick

This comment has been minimized.

Copy link

dswarbrick commented Jul 2, 2018

I have two drop-downs / variables, executing the label_values() function to populate $instance and $cpu. The panel query is as follows:

rate(node_cpu_seconds_total{instance="$instance", cpu="$cpu", mode!="idle"}[1m])

For a single value of $instance, my dashboard repeats the panel for every value of $cpu

@davkal

This comment has been minimized.

Copy link
Contributor

davkal commented Jul 2, 2018

I cant repro this on master. Tried Firefox and Chrome. Digging a bit through the code...

@davkal

This comment has been minimized.

Copy link
Contributor

davkal commented Jul 2, 2018

Found a good repro:

  • robust perception data source
  • bar chart
  • query: prometheus_engine_query_duration_seconds > 0.0001
  • set times from 2018-07-02 13:01:30 to 2018-07-02 13:06:30

-> all normal (stats.timeStep is 15000)

  • set times from 2018-07-02 13:01:29 to 2018-07-02 13:06:29

-> slim bars (stats.timeStep is incorrectly 1000)

@dswarbrick

This comment has been minimized.

Copy link

dswarbrick commented Jul 2, 2018

@davkal I think it depends on what time you initially open the dashboard, which is why it seems a bit hit and miss. If you're lucky enough to hit a multiple of $__interval, everything is fine. As I mentioned earlier however, simply clicking in the manual dashboard refresh several times lets you see the whole spectrum of bar widths ;-)

@davkal

This comment has been minimized.

Copy link
Contributor

davkal commented Jul 2, 2018

Yup, but the whole point of alignment is that it should not matter when you open the dashboard. I'm debugging this now...

Found the culprit: the null filling is not aligned. Hence you get oddly stepped values injected. The step informs the bar widths. If the injected values are close to real values, the step is really small and the chart lib thinks there are lots of bars and sets a small width.

@torkelo torkelo added this to the 5.3 milestone Jul 2, 2018

marefr added a commit that referenced this issue Jul 3, 2018

@marefr marefr modified the milestones: 5.3, 5.2.2 Jul 6, 2018

marefr added a commit that referenced this issue Jul 6, 2018

ryantxu added a commit to ryantxu/grafana that referenced this issue Jul 8, 2018

Merge remote-tracking branch 'grafana/master' into lazy-load-panels
* grafana/master: (78 commits)
  devenv: updated devenv provision scripts
  ux: minor fix/tweak to inactive view mode, think logo should be visible & fixes dashboard title alignment
  changelog: add notes about closing grafana#12379
  Fix datasource sorting with template variables
  changelog: add notes about closing grafana#12484
  changelog: add notes about closing grafana#12506
  changelog: add notes about closing grafana#12506
  fix links not updating after changing variables
  prometheus heatmap: fix unhandled error when some points are missing
  fix caret for help button is ds http settings
  changelog: add notes about closing grafana#11618 grafana#11619
  Basic cleanup
  Add mocks in test file
  Remove q and stub
  Add Jest stubs
  Remove async
  Remove logs and comments
  Start elastic ds test conversion
  run enterprise build only on master for now
  refix the settings indentation
  ...
@FUSAKLA

This comment has been minimized.

Copy link

FUSAKLA commented Jul 18, 2018

Hi, is there any planned date of releasing this fix? I see that 5.2.2 milestone is complete.
Charts using bars are kind of useless right now, so just to know if I should rollback or wait.
Thanks for any info.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.