Incorrect graph for node_disk_io_now

[candlerb]

In the panel "Disk Detail" is a graph "Disk IOs Current In Progress". The query it uses is:

irate(node_disk_io_now{instance=~"$node:$port",job=~"$job"}[5m])

However, node_disk_io_now is not a counter, it's a gauge:

# HELP node_disk_io_now The number of I/Os currently in progress.
# TYPE node_disk_io_now gauge

The io_now value is field 9 documented here:

Field  9 -- # of I/Os currently in progress
    The only field that should go to zero. Incremented as requests are
    given to appropriate struct request_queue and decremented as they finish.

Therefore I believe the irate(...) wrapper needs to be removed. This is already an instantaneous snapshot of the outstanding I/O requests.

[rfrail3]

Thanks for pointing the issue, it's is fixed now.

[candlerb]

I just checked the latest install of 1860 (node-exporter-full_rev23.json), and the panel Storage Disk > Instantaneous Queue Size still shows:

rate(node_disk_io_now{instance="$node",job="$job"}[$__rate_interval])

I believe it should be instead:

node_disk_io_now{instance="$node",job="$job"}

In addition:

• The Y axis units are currently set to I/O ops/sec (iops); I think this should be Misc > none. The metric is the queue depth, so it's "number of outstanding requests", which is dimensionless and definitely not a rate of any kind.
• The Y axis label is set to "I/Os". This is acceptable, as it can be interpreted as "number of I/Os", rather than "I/Os per second". Maybe "Outstanding requests" would be better, but I don't mind too strongly.

The corrected panel would look like this:

This shows the number of active requests is either 0 or 1 at each sample time, which is correct.

[candlerb]

(Aside: I can prepare a PR for this, but I have other graphs I want to check first)

[candlerb]

Since I'm running Grafana 8.0.5, and the dashboard is for 7.3.7, the diffs turned out to be rather large, especially because large chunks of indentation have changed due to a change in panels/targets layout.

Probably best just to apply the two changes by hand:

-              "expr": "rate(node_disk_io_now{instance=\"$node\",job=\"$job\"}[$__rate_interval])",
+          "expr": "node_disk_io_now{instance=\"$node\",job=\"$job\"}",
...
-              "format": "iops",
+          "format": "none",

[rfrail3]

It was a regression on d52e0ec
Let me fix and check the other changes that you reported over the current dashboard.

Thanks Brian,