pcm-pcie.x result seems not right

[cooljiansir]

hi ! I tried pcm-pcie.x on my Intel(R) Xeon(R) CPU E5，but the result(read\write bandwidth) of pcm-pcie.x and iostat was very different：

$sudo ./pcm-pcie.x -B
Skt | PCIeRdCur |  RFO  |  CRd  |  DRd  |  ItoM  |  PRd  |  WiL  | PCIe Rd (B) | PCIe Wr (B)
 0       0           0     155 M   213 M      0       0      30 K          23 G           0
 1     140 K        15 K    26 K   352 K     31 M     0       0            34 M        1986 M
----------------------------------------------------------------------------------------------------
 *     140 K        15 K   155 M   214 M     31 M     0      30 K          23 G        1986 M

$iostat -mx 1
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme2n1           0.00     0.00 15554.00    0.00  1860.19     0.00   244.93   209.61   13.42   13.42    0.00   0.06 100.20

Then I tried on another Intel(R) Xeon(R) Platinum, the result(read\write bandwidth) was very different too:

$sudo ./pcm-pcie.x -B
Skt | PCIeRdCur |  RFO  |  CRd  |  DRd  |  ItoM  |  PRd  |  WiL  | PCIe Rd (B) | PCIe Wr (B)
 0       0           0       0       0        0       0       0             0             0
 1      65 K      7644       0       0      312       0       0          4706 K         509 K
----------------------------------------------------------------------------------------------------
 *      65 K      7644       0       0      312       0       0          4706 K         509 K

$iostat -mx 1
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme10n1          0.00     0.00 8012.00    0.00   916.97     0.00   234.39     1.28    0.16    0.16    0.00   0.10  81.60

[xerothermic]

Hi @cooljiansir,

Thanks for reporting. I see two issues to be addressed in pcm-pcie program.

1. Xeon E5 based machine (prior to Skylake), the PCIe Rd (B) column is reporting wrong number.
2. Xeon Platinum based machine, the stats is off for both read and write.

To answer 1) Currently, PCIe Rd (B) is estimating bandwidth using sum(PCIeRdCur + CRd + DRd) * 64. CRd and DRd are generally traffics coming from CPU (sometimes service IO requests when BIOS changes PCIe Read to use CRd or DRd coherent flow). We will drop the CRd and DRd from PCIe Rd bandwidth calculation, so CPU traffic will not be estimated in the future.

To answer 2) I think there is a bug that skylake version of pcm-pcie.x is reporting "DDIO miss" bandwidth only by default. Can you please try ./pcm-pcie.x -e and see if you see most of traffic? We'll fix the default bandwidth display in the next release.

[cooljiansir]

Thanks for your reply : )
I tried with ./pcm-pcie.x -e:(Intel(R) Xeon(R) Platinum)

$sudo ./pcm-pcie.x -e
Skt | PCIeRdCur |  RFO  |  CRd  |  DRd  |  ItoM  |  PRd  |  WiL
 0       0           0       0       0     2160       0       0  	(Total)
 0      17 K      1956       0       0     3987 K    24      12  	(Miss)
 0       0           0       0       0        0       0       0  	(Hit)
 1       0           0       0       0        0       0       0  	(Total)
 1       0           0       0       0        0       0    4668  	(Miss)
 1       0           0       0       0        0       0       0  	(Hit)
-----------------------------------------------------------------------
 *       0           0       0       0     2160       0       0  	(Aggregate)

$iostat -mx 1
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme1n1           0.00     0.00 2413.00    0.00   245.23     0.00   208.14     0.34    0.14    0.14    0.00   0.11  27.60

[cooljiansir]

@xerothermic I found that ItoM (Total) is smaller than ItoM (Miss), I'm wondering if we should add the result of PCM::programPCIeMissCounters and PCM::programPCIeCounters?

[xerothermic]

Hi zhijian, Yes, that is correct. Would you mind to submit a PR if you think you can do it? Thanks, Patrick

…

On Apr 8, 2018, at 01:50, zhijian ***@***.***> wrote: @xerothermic I found that ItoM (Total) is smaller than ItoM (Miss), I'm wondering if we should add the result of PCM::programPCIeMissCounters and PCM::programPCIeCounters? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[cooljiansir]

hi @xerothermic，I have another question about the hit and miss: on some of my sever, miss counter seems to include hit counter, but some other sever not.

So how can we get the total bandwidth?(hit + miss or max(hit, miss)?)