%rrqm and %wrqm values and which drive /proc or /sys metrics are used to calculate these?

[tomkcpr]

Hey There!

I've been trying to understand the meaning of %rrqm and %wrqm in relation to some behavior I'm seeing with slow RAID 6 resync speeds. The columns are maxing out near 100% for each drive, which by definition appears good, however values are marked red, stay near 100% which appears to indicate saturation. Yet it isn't clear from the description or the behavior where in the OS can adjustments be made to alleviate this. I've began to dig into the sysstat code to get some insight as to where the issue may be however haven't made much headway yet.

Wondering if I could get a bit more info on what OS values these numbers use to calculate the %rrqm and %wrqm and perhaps indicate what tunable directly affect these?

Cheers,

10/10/23 21:01:40
Device            r/s     rMB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wMB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dMB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
dm-0             0.00      0.00     0.00   0.00    0.00     0.00    0.90      0.03     0.00   0.00    3.33    32.94    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.12
dm-1             0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-2             0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
md127            0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
nvme0n1          0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
nvme1n1          0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
nvme2n1          0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sda              0.00      0.00     0.00   0.00    0.00     0.00    0.80      0.03     0.10  11.11    3.50    37.06    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.12
sdb             27.30     16.20  4120.00  99.34   15.97   607.66   39.90      2.35   561.90  93.37    2.10    60.32    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.52  13.39
sdc             25.40     16.20  4121.90  99.39   25.25   653.12   39.60      2.35   562.20  93.42    8.87    60.78    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.99  27.26
sdd             27.30     16.20  4120.00  99.34   16.93   607.66   39.70      2.35   562.10  93.40    2.77    60.63    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.57  14.23
sde             27.30     16.20  4120.00  99.34   16.12   607.66   39.80      2.35   562.00  93.39    2.52    60.47    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.54  14.10
sdf             23.40     16.20  4123.90  99.44   46.44   708.94   39.00      2.35   562.80  93.52   92.26    61.71    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    4.68  95.10
sdg             27.50     16.20  4119.80  99.34   16.60   603.24   42.50      4.70  1160.90  96.47    2.21   113.25    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.55  13.90

[tomkcpr]

Disregard pls. After disabling write and read merges, IO, throughput dropped significantly. Though would suggest perhaps mark near 100% with green and near 0% with red perhaps?