Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No perfctr data for MEM_DP (Fortran) #327

Closed
ErikP94 opened this issue Sep 28, 2020 · 27 comments
Closed

No perfctr data for MEM_DP (Fortran) #327

ErikP94 opened this issue Sep 28, 2020 · 27 comments

Comments

@ErikP94
Copy link

ErikP94 commented Sep 28, 2020

Hello,

I'm trying to get an overview of arithmetic and main memory performance of my program using the MEM_DP group in likwid-perfctr.
Unfortunately all the memory counters just show 0:

--------------------------------------------------------------------------------
CPU name:	Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
CPU type:	Intel Kabylake processor
CPU clock:	2.30 GHz
--------------------------------------------------------------------------------
 FLOPS/s =    243.49446939835948      MFLOPS/s
 FLOPS =    771147000.00000000      nflops
 Time =    3.1670000553131104      s
 Datasize =    158.18400000000000       MB
 Loads =    1542294000.0000000     
--------------------------------------------------------------------------------
Region likwid_compute, Group 1: MEM_DP
+-------------------+----------+
|    Region Info    |  Core 0  |
+-------------------+----------+
| RDTSC Runtime [s] | 3.346748 |
|     call count    |        1 |
+-------------------+----------+

+------------------------------------------+---------+-------------+
|                   Event                  | Counter |    Core 0   |
+------------------------------------------+---------+-------------+
|             INSTR_RETIRED_ANY            |  FIXC0  | 48479620000 |
|           CPU_CLK_UNHALTED_CORE          |  FIXC1  | 13018140000 |
|           CPU_CLK_UNHALTED_REF           |  FIXC2  |  7591400000 |
|              PWR_PKG_ENERGY              |   PWR0  |     52.1849 |
|              PWR_DRAM_ENERGY             |   PWR3  |      3.2621 |
| FP_ARITH_INST_RETIRED_128B_PACKED_DOUBLE |   PMC0  |           0 |
|    FP_ARITH_INST_RETIRED_SCALAR_DOUBLE   |   PMC1  |   800809100 |
| FP_ARITH_INST_RETIRED_256B_PACKED_DOUBLE |   PMC2  |           0 |
|                DRAM_READS                | MBOX0C1 |      -      |
|                DRAM_WRITES               | MBOX0C2 |      -      |
+------------------------------------------+---------+-------------+

+-----------------------------------+-----------+
|               Metric              |   Core 0  |
+-----------------------------------+-----------+
|        Runtime (RDTSC) [s]        |    3.3467 |
|        Runtime unhalted [s]       |    5.6503 |
|            Clock [MHz]            | 3950.9693 |
|                CPI                |    0.2685 |
|             Energy [J]            |   52.1849 |
|             Power [W]             |   15.5927 |
|          Energy DRAM [J]          |    3.2621 |
|           Power DRAM [W]          |    0.9747 |
|            DP [MFLOP/s]           |  239.2798 |
|          AVX DP [MFLOP/s]         |         0 |
|          Packed [MUOPS/s]         |         0 |
|          Scalar [MUOPS/s]         |  239.2798 |
|  Memory load bandwidth [MBytes/s] |         0 |
|  Memory load data volume [GBytes] |         0 |
| Memory evict bandwidth [MBytes/s] |         0 |
| Memory evict data volume [GBytes] |         0 |
|    Memory bandwidth [MBytes/s]    |         0 |
|    Memory data volume [GBytes]    |         0 |
|       Operational intensity       |    inf    |
+-----------------------------------+-----------+

I get the same results with and without the Marker API.
My testprogram is written in Fortran, I used the
gcc (Ubuntu 9.3.0-10ubuntu2) 9.3.0
to compile likwid.

The Fortran Module is compiled with gfortran (same version as gcc naturally). The changes in the config.mk are the following:

COMPILER = GCC#NO SPACE

# Path were to install likwid
PREFIX ?= /usr/local/likwid#NO SPACE

ACCESSMODE = perf_event#NO SPACE

FORTRAN_INTERFACE = true#NO SPACE

The changes in the make/inclue_GCC.mk are the following:

FC  = gfortran

FCFLAGS  = -fmodules ./  # gfortran

The command I run is:

sudo likwid-perfctr -C 0 -g MEM_DP ./likwid_testprogram

I hope I haven't overlooked anything and I'm sorry if I did!
Hope you can help me out, thanks in advance!

Erik
@TomTheBear
Copy link
Member

Hi Erik,

it looks like LIKWID cannot open the memory controler device. Does your system provide /sys/devices/uncore_imc? likwid-perfctr outputs - for non-accessible devices and counters. You should see a message when you increase the verbosity level -V 1. If there is no message, please supply the output of sudo likwid-perfctr -C 0 -g MEM_DP -V 3 ./likwid_testprogram. Using LIKWID with sudo sometimes causes problems. If your perf_event_paranoid setting is 0, you don't need sudo for LIKWID.

Not sure if this might be a problem, your selection of the FCFLAGS is not what is recommended for LIKWID. The make/include_GCC.mk contains this commented out line for gfortran.

Best,
Thomas

@ErikP94
Copy link
Author

ErikP94 commented Sep 28, 2020

Hi Thomas,

thanks for your quick reply!
I in fact don't see the the /sys/devices/uncore_imc. I just have /sys/devices/uncore_arb/ and /sys/devices/uncore_cbox_0/ - /sys/devices/uncore_cbox3.
I ran the command with the Verbose=3 option and get the following output:

--------------------------------------------------------------------------------
CPU name:	Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
CPU type:	Intel Kabylake processor
CPU clock:	2.30 GHz
CPU family:	6
CPU model:	142
CPU short:	skylake
CPU stepping:	12
CPU features:	FP ACPI MMX SSE SSE2 HTT TM RDTSCP MONITOR VMX EIST TM2 SSSE FMA SSE4.1 SSE4.2 AES AVX RDRAND AVX2 RDSEED SSE3 
CPU arch:	x86_64
--------------------------------------------------------------------------------
PERFMON version:	4
PERFMON number of counters:	4
PERFMON width of counters:	48
PERFMON number of fixed counters:	3
--------------------------------------------------------------------------------
[likwid-pin] Main PID -> core 0 - OK
Executing: ./likwid_testprogram
DEBUG - [perfmon_addEventSet:1983] Currently 1 groups of 2 active
DEBUG - [perfgroup_readGroup:852] Reading group MEM_DP from /usr/local/share/likwid/perfgroups/skylake/MEM_DP.txt
DEBUG - [perfmon_addEventSet:2162] Added event INSTR_RETIRED_ANY for counter FIXC0 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event CPU_CLK_UNHALTED_CORE for counter FIXC1 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event CPU_CLK_UNHALTED_REF for counter FIXC2 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event PWR_PKG_ENERGY for counter PWR0 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event PWR_DRAM_ENERGY for counter PWR3 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event FP_ARITH_INST_RETIRED_128B_PACKED_DOUBLE for counter PMC0 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event FP_ARITH_INST_RETIRED_SCALAR_DOUBLE for counter PMC1 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event FP_ARITH_INST_RETIRED_256B_PACKED_DOUBLE for counter PMC2 to group 0
DEBUG - [perfmon_addEventSet:2102] Cannot access counter register MBOX0C1
DEBUG - [perfmon_addEventSet:2102] Cannot access counter register MBOX0C2
DEBUG - [perfmon_setupCountersThread_perfevent:634] SETUP_FIXED [0] Register 0x0 , Flags: 0x1 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=140400 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:634] SETUP_FIXED [0] Register 0x1 , Flags: 0x0 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=140400 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:634] SETUP_FIXED [0] Register 0x2 , Flags: 0x9 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=140400 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:644] SETUP_POWER [0] Register 0xD , Flags: 0x0 
DEBUG - [perf_uncore_setup:484] Get information for uncore counters from folder /sys/bus/event_source/devices/power
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:644] SETUP_POWER [0] Register 0x10 , Flags: 0x0 
DEBUG - [perf_uncore_setup:484] Get information for uncore counters from folder /sys/bus/event_source/devices/power
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:638] SETUP_PMC [0] Register 0x3 , Flags: 0x4C7 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=140400 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:638] SETUP_PMC [0] Register 0x4 , Flags: 0x1C7 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=140400 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:638] SETUP_PMC [0] Register 0x5 , Flags: 0x10C7 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=140400 flags=0
--------------------------------------------------------------------------------
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x3 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x3 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x3 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x4 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x4 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x4 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x5 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x5 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x5 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x6 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x6 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x6 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x7 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x7 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x7 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x8 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x8 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x8 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x9 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x9 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x9 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0xA , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0xA , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0xA , Flags: 0x0 
Running without Marker API. Activate Marker API with -m on commandline.
 FLOPS/s =    259.47073628600651      MFLOPS/s
 FLOPS =    771147000.00000000      nflops
 Time =    2.9719998836517334      s
 Datasize =    158.18400000000000       MB
 Loads =    1542294000.0000000     
DEBUG - [perfmon_stopCountersThread_perfevent:824] FREEZE_COUNTER [0] Register 0x3 , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:828] READ_COUNTER [0] Register 0x3 , Flags: 0xB49ADC673 
DEBUG - [perfmon_stopCountersThread_perfevent:834] RESET_COUNTER [0] Register 0x3 , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:824] FREEZE_COUNTER [0] Register 0x4 , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:828] READ_COUNTER [0] Register 0x4 , Flags: 0x306E326F7 
DEBUG - [perfmon_stopCountersThread_perfevent:834] RESET_COUNTER [0] Register 0x4 , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:824] FREEZE_COUNTER [0] Register 0x5 , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:828] READ_COUNTER [0] Register 0x5 , Flags: 0x1AB1D9420 
DEBUG - [perfmon_stopCountersThread_perfevent:834] RESET_COUNTER [0] Register 0x5 , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:824] FREEZE_COUNTER [0] Register 0x6 , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:828] READ_COUNTER [0] Register 0x6 , Flags: 0x3379C40000 
DEBUG - [perfmon_stopCountersThread_perfevent:834] RESET_COUNTER [0] Register 0x6 , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:824] FREEZE_COUNTER [0] Register 0x7 , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:828] READ_COUNTER [0] Register 0x7 , Flags: 0x115B40000 
DEBUG - [perfmon_stopCountersThread_perfevent:834] RESET_COUNTER [0] Register 0x7 , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:824] FREEZE_COUNTER [0] Register 0x8 , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:828] READ_COUNTER [0] Register 0x8 , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:834] RESET_COUNTER [0] Register 0x8 , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:824] FREEZE_COUNTER [0] Register 0x9 , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:828] READ_COUNTER [0] Register 0x9 , Flags: 0x2FBB4E3D 
DEBUG - [perfmon_stopCountersThread_perfevent:834] RESET_COUNTER [0] Register 0x9 , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:824] FREEZE_COUNTER [0] Register 0xA , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:828] READ_COUNTER [0] Register 0xA , Flags: 0x0 
DEBUG - [perfmon_stopCountersThread_perfevent:834] RESET_COUNTER [0] Register 0xA , Flags: 0x0 
--------------------------------------------------------------------------------
Group 1: MEM_DP
+------------------------------------------+---------+-------------+
|                   Event                  | Counter |    Core 0   |
+------------------------------------------+---------+-------------+
|             INSTR_RETIRED_ANY            |  FIXC0  | 48480765555 |
|           CPU_CLK_UNHALTED_CORE          |  FIXC1  | 13000451831 |
|           CPU_CLK_UNHALTED_REF           |  FIXC2  |  7165809696 |
|              PWR_PKG_ENERGY              |   PWR0  |     51.4756 |
|              PWR_DRAM_ENERGY             |   PWR3  |      1.0848 |
| FP_ARITH_INST_RETIRED_128B_PACKED_DOUBLE |   PMC0  |           0 |
|    FP_ARITH_INST_RETIRED_SCALAR_DOUBLE   |   PMC1  |   800804413 |
| FP_ARITH_INST_RETIRED_256B_PACKED_DOUBLE |   PMC2  |           0 |
|                DRAM_READS                | MBOX0C1 |      -      |
|                DRAM_WRITES               | MBOX0C2 |      -      |
+------------------------------------------+---------+-------------+

+-----------------------------------+-----------+
|               Metric              |   Core 0  |
+-----------------------------------+-----------+
|        Runtime (RDTSC) [s]        |    3.1681 |
|        Runtime unhalted [s]       |    5.6426 |
|            Clock [MHz]            | 4179.9288 |
|                CPI                |    0.2682 |
|             Energy [J]            |   51.4756 |
|             Power [W]             |   16.2481 |
|          Energy DRAM [J]          |    1.0848 |
|           Power DRAM [W]          |    0.3424 |
|            DP [MFLOP/s]           |  252.7712 |
|          AVX DP [MFLOP/s]         |         0 |
|          Packed [MUOPS/s]         |         0 |
|          Scalar [MUOPS/s]         |  252.7712 |
|  Memory load bandwidth [MBytes/s] |         0 |
|  Memory load data volume [GBytes] |         0 |
| Memory evict bandwidth [MBytes/s] |         0 |
| Memory evict data volume [GBytes] |         0 |
|    Memory bandwidth [MBytes/s]    |         0 |
|    Memory data volume [GBytes]    |         0 |
|       Operational intensity       |    inf    |
+-----------------------------------+-----------+

I see that there seems to be a problem:

DEBUG - [perfmon_addEventSet:2102] Cannot access counter register MBOX0C1
DEBUG - [perfmon_addEventSet:2102] Cannot access counter register MBOX0C2

but I don't know how to solve it.

As for the sudo: I will try changing my perf_event_paranoid to 0, but since I have to reboot the system after the change (I think), I will have to wait until I finish some things on my computer first.

Changing the FCFLAGS on my own was just a stupid mistake, I simply overlooked the commented Flags for gfortran. Thanks alot for that hint! I ran make distclean && make && sudo make install, but the problem remained.

Thanks again for your help!

Best,
Erik

@TomTheBear
Copy link
Member

Hi Erik,

there is no way to solve it with the perf_event backend despite installing a newer kernel version with the hope that this version provides the device. If you switch the ACCESSMODE to accessdaemon and install LIKWID with make distclean && make && sudo make install, it uses the default way of accessing the counters. This works without special kernel modules, just the msr module (sudo modprobe msr) and does not require perf_event.

You can configure your system to set paranoid level to 0 at boot (add kernel.perf_event_paranoid=0 to /etc/sysctl.conf).

Best,
Thomas

@ErikP94
Copy link
Author

ErikP94 commented Sep 28, 2020

You can configure your system to set paranoid level to 0 at boot (add kernel.perf_event_paranoid=0 to /etc/sysctl.conf).

I did that, so I don't have to use sudo anymore. Thanks for that.

I also changed the ACCESSMODE to accessdeamon and reinstalled Likwid. I also added msr to the Linux kernel. Still the same output.
So am I basically unable to count those certain memory events on my machine?

Best,
Erik

@TomTheBear
Copy link
Member

Your setting in /etc/sysctl.conf works only at the next boot. To do it right now, you can do sudo sysctl kernel.perf_event_paranoid=0.

Please supply another output of your run. The backends behave completely different and the output should show something. The accessdaemon basically opens /dev/mem and accesses the memory at specific offsets.

Best,
Thomas

@ErikP94
Copy link
Author

ErikP94 commented Sep 29, 2020

Hi Thomas

if I run:

likwid-perfctr -m -C 0 -g MEM_DP -V 3 ./likwid_testprogram

I get the following output:

--------------------------------------------------------------------------------
CPU name:	Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
CPU type:	Intel Kabylake processor
CPU clock:	2.30 GHz
CPU family:	6
CPU model:	142
CPU short:	skylake
CPU stepping:	12
CPU features:	FP ACPI MMX SSE SSE2 HTT TM RDTSCP MONITOR VMX EIST TM2 SSSE FMA SSE4.1 SSE4.2 AES AVX RDRAND AVX2 RDSEED SSE3 
CPU arch:	x86_64
--------------------------------------------------------------------------------
PERFMON version:	4
PERFMON number of counters:	4
PERFMON width of counters:	48
PERFMON number of fixed counters:	3
--------------------------------------------------------------------------------
[likwid-pin] Main PID -> core 0 - OK
Executing: ./likwid_testprogram
DEBUG - [perfmon_addEventSet:1983] Currently 1 groups of 2 active
DEBUG - [perfgroup_readGroup:852] Reading group MEM_DP from /usr/local/share/likwid/perfgroups/skylake/MEM_DP.txt
DEBUG - [perfmon_addEventSet:2162] Added event INSTR_RETIRED_ANY for counter FIXC0 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event CPU_CLK_UNHALTED_CORE for counter FIXC1 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event CPU_CLK_UNHALTED_REF for counter FIXC2 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event PWR_PKG_ENERGY for counter PWR0 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event PWR_DRAM_ENERGY for counter PWR3 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event FP_ARITH_INST_RETIRED_128B_PACKED_DOUBLE for counter PMC0 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event FP_ARITH_INST_RETIRED_SCALAR_DOUBLE for counter PMC1 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event FP_ARITH_INST_RETIRED_256B_PACKED_DOUBLE for counter PMC2 to group 0
DEBUG - [perfmon_addEventSet:2102] Cannot access counter register MBOX0C1
DEBUG - [perfmon_addEventSet:2102] Cannot access counter register MBOX0C2
DEBUG - [perfmon_setupCountersThread_perfevent:634] SETUP_FIXED [0] Register 0x0 , Flags: 0x1 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=6543 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:634] SETUP_FIXED [0] Register 0x1 , Flags: 0x0 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=6543 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:634] SETUP_FIXED [0] Register 0x2 , Flags: 0x9 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=6543 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:644] SETUP_POWER [0] Register 0xD , Flags: 0x0 
DEBUG - [perf_uncore_setup:484] Get information for uncore counters from folder /sys/bus/event_source/devices/power
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:644] SETUP_POWER [0] Register 0x10 , Flags: 0x0 
DEBUG - [perf_uncore_setup:484] Get information for uncore counters from folder /sys/bus/event_source/devices/power
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:638] SETUP_PMC [0] Register 0x3 , Flags: 0x4C7 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=6543 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:638] SETUP_PMC [0] Register 0x4 , Flags: 0x1C7 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=6543 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:638] SETUP_PMC [0] Register 0x5 , Flags: 0x10C7 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=6543 flags=0
--------------------------------------------------------------------------------
DEBUG - [perfmon_addEventSet:1983] Currently 1 groups of 2 active
DEBUG - [perfgroup_readGroup:852] Reading group MEM_DP from /usr/local/share/likwid/perfgroups/skylake/MEM_DP.txt
DEBUG - [perfmon_addEventSet:2162] Added event INSTR_RETIRED_ANY for counter FIXC0 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event CPU_CLK_UNHALTED_CORE for counter FIXC1 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event CPU_CLK_UNHALTED_REF for counter FIXC2 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event PWR_PKG_ENERGY for counter PWR0 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event PWR_DRAM_ENERGY for counter PWR3 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event FP_ARITH_INST_RETIRED_128B_PACKED_DOUBLE for counter PMC0 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event FP_ARITH_INST_RETIRED_SCALAR_DOUBLE for counter PMC1 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event FP_ARITH_INST_RETIRED_256B_PACKED_DOUBLE for counter PMC2 to group 0
DEBUG - [perfmon_addEventSet:2102] Cannot access counter register MBOX0C1
DEBUG - [perfmon_addEventSet:2102] Cannot access counter register MBOX0C2
DEBUG - [perfmon_setupCountersThread_perfevent:634] SETUP_FIXED [0] Register 0x0 , Flags: 0x1 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=6543 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:634] SETUP_FIXED [0] Register 0x1 , Flags: 0x0 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=6543 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:634] SETUP_FIXED [0] Register 0x2 , Flags: 0x9 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=6543 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:644] SETUP_POWER [0] Register 0xD , Flags: 0x0 
DEBUG - [perf_uncore_setup:484] Get information for uncore counters from folder /sys/bus/event_source/devices/power
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:644] SETUP_POWER [0] Register 0x10 , Flags: 0x0 
DEBUG - [perf_uncore_setup:484] Get information for uncore counters from folder /sys/bus/event_source/devices/power
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:638] SETUP_PMC [0] Register 0x3 , Flags: 0x4C7 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=6543 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:638] SETUP_PMC [0] Register 0x4 , Flags: 0x1C7 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=6543 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:638] SETUP_PMC [0] Register 0x5 , Flags: 0x10C7 
DEBUG - [perfmon_setupCountersThread_perfevent:755] perf_event_open: cpu_id=0 pid=6543 flags=0
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:794] RESET_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_startCountersThread_perfevent:803] START_COUNTER [0] Register 0x0 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x5 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x5 , Flags: 0x11EB4 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x5 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x6 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x6 , Flags: 0x12F9C 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x6 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x7 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x7 , Flags: 0x3AC20 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x7 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x8 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x8 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x8 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x9 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x9 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x9 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0xA , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0xA , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0xA , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0xB , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0xB , Flags: 0x2 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0xB , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0xC , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0xC , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0xC , Flags: 0x0 
DEBUG - [likwid_markerStartRegion:556] START [likwid_compute] READ EVENT [0=0] EVENT 0 VALUE 73396
DEBUG - [likwid_markerStartRegion:556] START [likwid_compute] READ EVENT [0=0] EVENT 1 VALUE 77724
DEBUG - [likwid_markerStartRegion:556] START [likwid_compute] READ EVENT [0=0] EVENT 2 VALUE 240672
DEBUG - [likwid_markerStartRegion:556] START [likwid_compute] READ EVENT [0=0] EVENT 3 VALUE 0
DEBUG - [likwid_markerStartRegion:556] START [likwid_compute] READ EVENT [0=0] EVENT 4 VALUE 0
DEBUG - [likwid_markerStartRegion:556] START [likwid_compute] READ EVENT [0=0] EVENT 5 VALUE 0
DEBUG - [likwid_markerStartRegion:556] START [likwid_compute] READ EVENT [0=0] EVENT 6 VALUE 2
DEBUG - [likwid_markerStartRegion:556] START [likwid_compute] READ EVENT [0=0] EVENT 7 VALUE 0
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x5 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x5 , Flags: 0xB499F908C 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x5 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x6 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x6 , Flags: 0x32E034B60 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x6 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x7 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x7 , Flags: 0x1C3B10BC0 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x7 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x8 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x8 , Flags: 0x37435C0000 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x8 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0x9 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0x9 , Flags: 0x1071C0000 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0x9 , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0xA , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0xA , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0xA , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0xB , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0xB , Flags: 0x2FBB6C3B 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0xB , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:852] FREEZE_COUNTER [0] Register 0xC , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:856] READ_COUNTER [0] Register 0xC , Flags: 0x0 
DEBUG - [perfmon_readCountersThread_perfevent:861] UNFREEZE_COUNTER [0] Register 0xC , Flags: 0x0 
DEBUG - [likwid_markerStopRegion:625] STOP [likwid_compute] READ EVENT [0=0] EVENT 0 VALUE 48479834252
DEBUG - [likwid_markerStopRegion:625] STOP [likwid_compute] READ EVENT [0=0] EVENT 1 VALUE 13656869728
DEBUG - [likwid_markerStopRegion:625] STOP [likwid_compute] READ EVENT [0=0] EVENT 2 VALUE 7578127296
DEBUG - [likwid_markerStopRegion:625] STOP [likwid_compute] READ EVENT [0=0] EVENT 3 VALUE 237353304064
DEBUG - [likwid_markerStopRegion:625] STOP [likwid_compute] READ EVENT [0=0] EVENT 4 VALUE 4414242816
DEBUG - [likwid_markerStopRegion:625] STOP [likwid_compute] READ EVENT [0=0] EVENT 5 VALUE 0
DEBUG - [likwid_markerStopRegion:625] STOP [likwid_compute] READ EVENT [0=0] EVENT 6 VALUE 800812091
DEBUG - [likwid_markerStopRegion:625] STOP [likwid_compute] READ EVENT [0=0] EVENT 7 VALUE 0
 FLOPS/s =    243.80239393365028      MFLOPS/s
 FLOPS =    771147000.00000000      nflops
 Time =    3.1630001068115234      s
 Datasize =    158.18400000000000       MB
DEBUG - [likwid_markerClose:417] Creating Marker file /tmp/likwid_6536.txt with 1 regions 1 groups and 1 threads
DEBUG - [likwid_markerClose:422] 1 1 1
DEBUG - [likwid_markerClose:431] 0:likwid_compute-0
DEBUG - [likwid_markerClose:457] 0 0 0 1 3.322743e+00 10 4.847976e+10 1.365679e+10 7.577887e+09 5.526312e+01 1.027771e+00 0.000000e+00 8.008121e+08 0.000000e+00 nan nan 
--------------------------------------------------------------------------------
Region likwid_compute, Group 1: MEM_DP
+-------------------+----------+
|    Region Info    |  Core 0  |
+-------------------+----------+
| RDTSC Runtime [s] | 3.322743 |
|     call count    |        1 |
+-------------------+----------+

+------------------------------------------+---------+-------------+
|                   Event                  | Counter |    Core 0   |
+------------------------------------------+---------+-------------+
|             INSTR_RETIRED_ANY            |  FIXC0  | 48479760000 |
|           CPU_CLK_UNHALTED_CORE          |  FIXC1  | 13656790000 |
|           CPU_CLK_UNHALTED_REF           |  FIXC2  |  7577887000 |
|              PWR_PKG_ENERGY              |   PWR0  |     55.2631 |
|              PWR_DRAM_ENERGY             |   PWR3  |      1.0278 |
| FP_ARITH_INST_RETIRED_128B_PACKED_DOUBLE |   PMC0  |           0 |
|    FP_ARITH_INST_RETIRED_SCALAR_DOUBLE   |   PMC1  |   800812100 |
| FP_ARITH_INST_RETIRED_256B_PACKED_DOUBLE |   PMC2  |           0 |
|                DRAM_READS                | MBOX0C1 |      -      |
|                DRAM_WRITES               | MBOX0C2 |      -      |
+------------------------------------------+---------+-------------+

+-----------------------------------+-----------+
|               Metric              |   Core 0  |
+-----------------------------------+-----------+
|        Runtime (RDTSC) [s]        |    3.3227 |
|        Runtime unhalted [s]       |    5.9275 |
|            Clock [MHz]            | 4152.2119 |
|                CPI                |    0.2817 |
|             Energy [J]            |   55.2631 |
|             Power [W]             |   16.6318 |
|          Energy DRAM [J]          |    1.0278 |
|           Power DRAM [W]          |    0.3093 |
|            DP [MFLOP/s]           |  241.0093 |
|          AVX DP [MFLOP/s]         |         0 |
|          Packed [MUOPS/s]         |         0 |
|          Scalar [MUOPS/s]         |  241.0093 |
|  Memory load bandwidth [MBytes/s] |         0 |
|  Memory load data volume [GBytes] |         0 |
| Memory evict bandwidth [MBytes/s] |         0 |
| Memory evict data volume [GBytes] |         0 |
|    Memory bandwidth [MBytes/s]    |         0 |
|    Memory data volume [GBytes]    |         0 |
|       Operational intensity       |    inf    |
+-----------------------------------+-----------+

It looks quite similar to the output with perf_event.
I made make distclean && make && sudo make install (and of course changed the config.mk before).

@TomTheBear
Copy link
Member

As you noticed, this is still using the perf_event backend. Maybe download a fresh copy, not sure why make distclean && make && sudo make install does not work. You could also try a sudo make uninstall first to clean you installation path.

@ErikP94
Copy link
Author

ErikP94 commented Sep 29, 2020 •
edited

Alright, sorry, it looked for the binaries in an older Installation Path. Now I've got the binaries of the freshly installed version.
First I got this output:

Cannot get access to MSRs. Please check permissions to the MSRs

After I ran sudo modprobe msr and retried, I got:

--------------------------------------------------------------------------------
CPU name:	Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
CPU type:	Intel Kabylake processor
CPU clock:	2.30 GHz
CPU family:	6
CPU model:	142
CPU short:	skylake
CPU stepping:	12
CPU features:	FP ACPI MMX SSE SSE2 HTT TM RDTSCP MONITOR VMX EIST TM2 SSSE FMA SSE4.1 SSE4.2 AES AVX RDRAND AVX2 RDSEED SSE3 
CPU arch:	x86_64
--------------------------------------------------------------------------------
PERFMON version:	4
PERFMON number of counters:	4
PERFMON width of counters:	48
PERFMON number of fixed counters:	3
--------------------------------------------------------------------------------
[likwid-pin] Main PID -> core 0 - OK
DEBUG - [HPMinit:98] Adjusting functions for x86 architecture in daemon mode
DEBUG - [access_client_startDaemon:137] Starting daemon /usr/local/sbin/likwid-accessD
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17491 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17491 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17491 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17491 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17491 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17491 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17491 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17491 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17491 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17491 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17491 for CPU 0...
DEBUG - [access_client_startDaemon:197] Successfully opened socket /tmp/likwid-17491 to daemon for CPU 0
DEBUG - [HPMaddThread:143] Adding CPU 0 to access module
DEBUG - [access_client_check:505] Device check for dev 5 on CPU 0 with accessDaemon failed: no such pci device

DEBUG - [access_client_check:505] Device check for dev 5 on CPU 0 with accessDaemon failed: no such pci device

DEBUG - [access_client_check:505] Device check for dev 5 on CPU 0 with accessDaemon failed: no such pci device

DEBUG - [access_client_check:505] Device check for dev 5 on CPU 0 with accessDaemon failed: no such pci device

DEBUG - [access_client_check:505] Device check for dev 5 on CPU 0 with accessDaemon failed: no such pci device

DEBUG - [access_client_read:331] Got error 'failed to read/write register' from access daemon reading reg 0x63B at CPU 0
DEBUG - [power_init:404] Deactivating perf register for RAPL domain PP0
DEBUG - [access_client_read:331] Got error 'failed to read/write register' from access daemon reading reg 0x61C at CPU 0
DEBUG - [power_init:378] Deactivating info register for RAPL domain DRAM
DEBUG - [access_client_write:421] Got error 'failed to read/write register' from access daemon writing reg 0x3F1 at CPU 0
ERROR - [./src/includes/perfmon_skylake.h:perfmon_init_skylake:72] No such file or directory.
Cannot zero 0x3F1 (0x3F1)
DEBUG - [access_client_write:421] Got error 'failed to read/write register' from access daemon writing reg 0x3F7 at CPU 0
ERROR - [./src/includes/perfmon_skylake.h:perfmon_init_skylake:77] No such file or directory.
Cannot zero 0x3F7 (0x3F7)
Executing: ./likwid_testprogram
DEBUG - [perfmon_addEventSet:1983] Currently 1 groups of 2 active
DEBUG - [perfgroup_readGroup:852] Reading group MEM_DP from /usr/local/likwid/share/likwid/perfgroups/skylake/MEM_DP.txt
DEBUG - [access_client_write:421] Got error 'failed to read/write register' from access daemon writing reg 0x38D at CPU 0
DEBUG - [checkAccess:267] Counter FIXC0 not writeable on this machine
DEBUG - [perfmon_addEventSet:2093] Cannot access counter register FIXC0
DEBUG - [access_client_write:421] Got error 'failed to read/write register' from access daemon writing reg 0x38D at CPU 0
DEBUG - [checkAccess:267] Counter FIXC1 not writeable on this machine
DEBUG - [perfmon_addEventSet:2093] Cannot access counter register FIXC1
DEBUG - [access_client_write:421] Got error 'failed to read/write register' from access daemon writing reg 0x38D at CPU 0
DEBUG - [checkAccess:267] Counter FIXC2 not writeable on this machine
DEBUG - [perfmon_addEventSet:2093] Cannot access counter register FIXC2
DEBUG - [perfmon_addEventSet:2162] Added event PWR_PKG_ENERGY for counter PWR0 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event PWR_DRAM_ENERGY for counter PWR3 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event FP_ARITH_INST_RETIRED_128B_PACKED_DOUBLE for counter PMC0 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event FP_ARITH_INST_RETIRED_SCALAR_DOUBLE for counter PMC1 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event FP_ARITH_INST_RETIRED_256B_PACKED_DOUBLE for counter PMC2 to group 0
DEBUG - [access_client_check:505] Device check for dev 5 on CPU 0 with accessDaemon failed: no such pci device

DEBUG - [checkAccess:228] WARNING: The device for counter MBOX0C1 does not exist
DEBUG - [perfmon_addEventSet:2093] Cannot access counter register MBOX0C1
DEBUG - [access_client_check:505] Device check for dev 5 on CPU 0 with accessDaemon failed: no such pci device

DEBUG - [checkAccess:228] WARNING: The device for counter MBOX0C2 does not exist
DEBUG - [perfmon_addEventSet:2093] Cannot access counter register MBOX0C2
DEBUG - [perfmon_setupCounterThread_skylake:773] FREEZE_PMC_AND_FIXED [0] Register 0x38F , Flags: 0x0 
DEBUG - [access_client_write:421] Got error 'failed to read/write register' from access daemon writing reg 0x38F at CPU 0
ERROR - [./src/includes/perfmon_skylake.h:perfmon_setupCounterThread_skylake:774] No such file or directory.
MSR write operation failed
--------------------------------------------------------------------------------
DEBUG - [HPMinit:98] Adjusting functions for x86 architecture in daemon mode
DEBUG - [access_client_startDaemon:137] Starting daemon /usr/local/sbin/likwid-accessD
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17508 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17508 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17508 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17508 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17508 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17508 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17508 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17508 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17508 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17508 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17508 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17508 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17508 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17508 for CPU 0...
DEBUG - [access_client_startDaemon:185] Still waiting for socket /tmp/likwid-17508 for CPU 0...
DEBUG - [access_client_startDaemon:197] Successfully opened socket /tmp/likwid-17508 to daemon for CPU 0
DEBUG - [HPMaddThread:143] Adding CPU 0 to access module
DEBUG - [access_client_check:505] Device check for dev 5 on CPU 0 with accessDaemon failed: no such pci device

DEBUG - [access_client_check:505] Device check for dev 5 on CPU 0 with accessDaemon failed: no such pci device

DEBUG - [access_client_check:505] Device check for dev 5 on CPU 0 with accessDaemon failed: no such pci device

DEBUG - [access_client_check:505] Device check for dev 5 on CPU 0 with accessDaemon failed: no such pci device

DEBUG - [access_client_check:505] Device check for dev 5 on CPU 0 with accessDaemon failed: no such pci device

DEBUG - [access_client_read:331] Got error 'failed to read/write register' from access daemon reading reg 0x63B at CPU 0
DEBUG - [power_init:404] Deactivating perf register for RAPL domain PP0
DEBUG - [access_client_read:331] Got error 'failed to read/write register' from access daemon reading reg 0x61C at CPU 0
DEBUG - [power_init:378] Deactivating info register for RAPL domain DRAM
DEBUG - [access_client_write:421] Got error 'failed to read/write register' from access daemon writing reg 0x3F1 at CPU 0
ERROR - [./src/includes/perfmon_skylake.h:perfmon_init_skylake:72] No such file or directory.
Cannot zero 0x3F1 (0x3F1)
DEBUG - [access_client_write:421] Got error 'failed to read/write register' from access daemon writing reg 0x3F7 at CPU 0
ERROR - [./src/includes/perfmon_skylake.h:perfmon_init_skylake:77] No such file or directory.
Cannot zero 0x3F7 (0x3F7)
DEBUG - [perfmon_addEventSet:1983] Currently 1 groups of 2 active
DEBUG - [perfgroup_readGroup:852] Reading group MEM_DP from /usr/local/likwid/share/likwid/perfgroups/skylake/MEM_DP.txt
DEBUG - [access_client_write:421] Got error 'failed to read/write register' from access daemon writing reg 0x38D at CPU 0
DEBUG - [checkAccess:267] Counter FIXC0 not writeable on this machine
DEBUG - [perfmon_addEventSet:2093] Cannot access counter register FIXC0
DEBUG - [access_client_write:421] Got error 'failed to read/write register' from access daemon writing reg 0x38D at CPU 0
DEBUG - [checkAccess:267] Counter FIXC1 not writeable on this machine
DEBUG - [perfmon_addEventSet:2093] Cannot access counter register FIXC1
DEBUG - [access_client_write:421] Got error 'failed to read/write register' from access daemon writing reg 0x38D at CPU 0
DEBUG - [checkAccess:267] Counter FIXC2 not writeable on this machine
DEBUG - [perfmon_addEventSet:2093] Cannot access counter register FIXC2
DEBUG - [perfmon_addEventSet:2162] Added event PWR_PKG_ENERGY for counter PWR0 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event PWR_DRAM_ENERGY for counter PWR3 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event FP_ARITH_INST_RETIRED_128B_PACKED_DOUBLE for counter PMC0 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event FP_ARITH_INST_RETIRED_SCALAR_DOUBLE for counter PMC1 to group 0
DEBUG - [perfmon_addEventSet:2162] Added event FP_ARITH_INST_RETIRED_256B_PACKED_DOUBLE for counter PMC2 to group 0
DEBUG - [access_client_check:505] Device check for dev 5 on CPU 0 with accessDaemon failed: no such pci device

DEBUG - [checkAccess:228] WARNING: The device for counter MBOX0C1 does not exist
DEBUG - [perfmon_addEventSet:2093] Cannot access counter register MBOX0C1
DEBUG - [access_client_check:505] Device check for dev 5 on CPU 0 with accessDaemon failed: no such pci device

DEBUG - [checkAccess:228] WARNING: The device for counter MBOX0C2 does not exist
DEBUG - [perfmon_addEventSet:2093] Cannot access counter register MBOX0C2
DEBUG - [perfmon_setupCounterThread_skylake:773] FREEZE_PMC_AND_FIXED [0] Register 0x38F , Flags: 0x0 
DEBUG - [access_client_write:421] Got error 'failed to read/write register' from access daemon writing reg 0x38F at CPU 0
ERROR - [./src/includes/perfmon_skylake.h:perfmon_setupCounterThread_skylake:774] No such file or directory.
MSR write operation failed
DEBUG - [perfmon_startCountersThread_skylake:987] START_POWER [0] Register 0x611 , Flags: 0x26B197FD 
DEBUG - [perfmon_startCountersThread_skylake:987] START_POWER [0] Register 0x619 , Flags: 0x7DC9E94 
DEBUG - [access_client_write:421] Got error 'failed to read/write register' from access daemon writing reg 0xC1 at CPU 0
ERROR - [./src/includes/perfmon_skylake.h:perfmon_startCountersThread_skylake:973] No such file or directory.
MSR write operation failed
DEBUG - [likwid_markerStartRegion:556] START [likwid_compute] READ EVENT [0=0] EVENT 3 VALUE 649172989
DEBUG - [likwid_markerStartRegion:556] START [likwid_compute] READ EVENT [0=0] EVENT 4 VALUE 131899028
DEBUG - [likwid_markerStartRegion:556] START [likwid_compute] READ EVENT [0=0] EVENT 5 VALUE 0
DEBUG - [likwid_markerStartRegion:556] START [likwid_compute] READ EVENT [0=0] EVENT 6 VALUE 0
DEBUG - [likwid_markerStartRegion:556] START [likwid_compute] READ EVENT [0=0] EVENT 7 VALUE 0
DEBUG - [likwid_markerStopRegion:625] STOP [likwid_compute] READ EVENT [0=0] EVENT 3 VALUE 649172989
DEBUG - [likwid_markerStopRegion:625] STOP [likwid_compute] READ EVENT [0=0] EVENT 4 VALUE 131899028
DEBUG - [likwid_markerStopRegion:625] STOP [likwid_compute] READ EVENT [0=0] EVENT 5 VALUE 0
DEBUG - [likwid_markerStopRegion:625] STOP [likwid_compute] READ EVENT [0=0] EVENT 6 VALUE 0
DEBUG - [likwid_markerStopRegion:625] STOP [likwid_compute] READ EVENT [0=0] EVENT 7 VALUE 0
 FLOPS/s =    240.83292363523134      MFLOPS/s
 FLOPS =    771147000.00000000      nflops
 Time =    3.2019999027252197      s
 Datasize =    158.18400000000000       MB
DEBUG - [likwid_markerClose:417] Creating Marker file /tmp/likwid_17486.txt with 1 regions 1 groups and 1 threads
DEBUG - [likwid_markerClose:422] 1 1 1
DEBUG - [likwid_markerClose:431] 0:likwid_compute-0
DEBUG - [likwid_markerClose:457] 0 0 0 1 3.373334e+00 10 nan nan nan 0.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00 nan nan 
DEBUG - [HPMfinalize:170] Removing CPU 0 from access module
--------------------------------------------------------------------------------
Region likwid_compute, Group 1: MEM_DP
+-------------------+----------+
|    Region Info    |  Core 0  |
+-------------------+----------+
| RDTSC Runtime [s] | 3.373334 |
|     call count    |        1 |
+-------------------+----------+

+------------------------------------------+---------+--------+
|                   Event                  | Counter | Core 0 |
+------------------------------------------+---------+--------+
|             INSTR_RETIRED_ANY            |  FIXC0  |    -   |
|           CPU_CLK_UNHALTED_CORE          |  FIXC1  |    -   |
|           CPU_CLK_UNHALTED_REF           |  FIXC2  |    -   |
|              PWR_PKG_ENERGY              |   PWR0  |      0 |
|              PWR_DRAM_ENERGY             |   PWR3  |      0 |
| FP_ARITH_INST_RETIRED_128B_PACKED_DOUBLE |   PMC0  |      0 |
|    FP_ARITH_INST_RETIRED_SCALAR_DOUBLE   |   PMC1  |      0 |
| FP_ARITH_INST_RETIRED_256B_PACKED_DOUBLE |   PMC2  |      0 |
|                DRAM_READS                | MBOX0C1 |    -   |
|                DRAM_WRITES               | MBOX0C2 |    -   |
+------------------------------------------+---------+--------+

+-----------------------------------+--------+
|               Metric              | Core 0 |
+-----------------------------------+--------+
|        Runtime (RDTSC) [s]        | 3.3733 |
|        Runtime unhalted [s]       |      0 |
|            Clock [MHz]            |    -   |
|                CPI                |    -   |
|             Energy [J]            |      0 |
|             Power [W]             |      0 |
|          Energy DRAM [J]          |      0 |
|           Power DRAM [W]          |      0 |
|            DP [MFLOP/s]           |      0 |
|          AVX DP [MFLOP/s]         |      0 |
|          Packed [MUOPS/s]         |      0 |
|          Scalar [MUOPS/s]         |      0 |
|  Memory load bandwidth [MBytes/s] |      0 |
|  Memory load data volume [GBytes] |      0 |
| Memory evict bandwidth [MBytes/s] |      0 |
| Memory evict data volume [GBytes] |      0 |
|    Memory bandwidth [MBytes/s]    |      0 |
|    Memory data volume [GBytes]    |      0 |
|       Operational intensity       |    -   |
+-----------------------------------+--------+

DEBUG - [perfmon_finalizeCountersThread_skylake:1766] SHOW_CTL [0] Device 0 Register 0x186 , Flags: 0x1104C7 
DEBUG - [perfmon_finalizeCountersThread_skylake:1768] CLEAR_CTL [0] Device 0 Register 0x186 , Flags: 0x0 
DEBUG - [access_client_write:421] Got error 'failed to read/write register' from access daemon writing reg 0x186 at CPU 0
ERROR - [./src/includes/perfmon_skylake.h:perfmon_finalizeCountersThread_skylake:1769] No such file or directory.
MSR write operation failed
DEBUG - [HPMfinalize:170] Removing CPU 0 from access module

Best,
Erik

@TomTheBear
Copy link
Member

It seems your system does not provide the memory controller counters. Those counters for desktop-style chips are not that well documented. It might be some system/BIOS setting but I'm not sure. I would recommend to use a server-style chip for the analysis.

DEBUG - [access_client_check:505] Device check for dev 5 on CPU 0 with accessDaemon failed: no such pci device
DEBUG - [checkAccess:228] WARNING: The device for counter MBOX0C1 does not exist
DEBUG - [perfmon_addEventSet:2093] Cannot access counter register MBOX0C1

Interestingly the fixed-purpose counters (FIXC0-2) don't seem to work as well. They are readable (checked first) but not writable. When I added the check I thought this case might never happen.

DEBUG - [access_client_write:421] Got error 'failed to read/write register' from access daemon writing reg 0x38D at CPU 0
DEBUG - [checkAccess:267] Counter FIXC0 not writeable on this machine
DEBUG - [perfmon_addEventSet:2093] Cannot access counter register FIXC0

It might be some daemon occupying the registers. We found one on RedHat systems which cause this problem (pmcd.service). Not sure about your kernel version but with version 5.9 writes to MSRs are forbidden (https://github.com/RRZE-HPC/likwid/wiki/likwid-perfctr#prerequisites-for-access-daemon-mode). The patch might be back-ported by your OS distribution.

@TomTheBear
Copy link
Member

I just found the Intel documentation of these memory controller counters in desktop chips: https://software.intel.com/content/www/us/en/develop/articles/monitoring-integrated-memory-controller-requests-in-the-2nd-3rd-and-4th-generation-intel.html

The page states only 2nd, 3rd, 4th, 5th and 6th generation. 6th generation is Skylake. Your chip uses the Comet Lake architecture. The link is Skylake -> Kabylake -> Coffeelake -> Comet Lake. I'm not sure whether Comet Lake is still covered by this "6th generation". But maybe the page wasn't properly updated (last update 2016).

@TomTheBear
Copy link
Member

Can you please try https://github.com/opcm/pcm on the system. It is developed by Intel and also provides these metrics. It should look like this.
image
It it works with PCM, the problem is on the LIKWID side.

@ErikP94
Copy link
Author

ErikP94 commented Oct 2, 2020

Thanks alot! I just cloned and built it. I will try it now.

@ErikP94
Copy link
Author

ErikP94 commented Oct 2, 2020

If i want to run e.g. pcm-memory I get the following output:

Processor Counter Monitor: Memory Bandwidth Monitoring Utility  ($Format:%ci ID=%h$)

 This utility measures memory bandwidth per channel or per DIMM rank in real-time

IBRS and IBPB supported  : yes
STIBP supported          : yes
Spec arch caps supported : yes
Number of physical cores: 4
Number of logical cores: 8
Number of online logical cores: 8
Threads (logical cores) per physical core: 2
Num sockets: 1
Physical cores per socket: 4
Core PMU (perfmon) version: 4
Number of core PMU generic (programmable) counters: 4
Width of generic (programmable) counters: 48 bits
Number of core PMU fixed counters: 3
Width of fixed counters: 48 bits
Nominal core frequency: 2300000000 Hz
IBRS enabled in the kernel   : yes
STIBP enabled in the kernel  : no
The processor is not susceptible to Rogue Data Cache Load: yes
The processor supports enhanced IBRS                     : yes
Package thermal spec power: 15 Watt; Package minimum power: 0 Watt; Package maximum power: 0 Watt;
opening /dev/mem failed: errno is 1 (Operation not permitted)
Can not read memory controller counter information from PCI configuration space. Access to memory bandwidth counters is not possible.
You must be root to access these SandyBridge/IvyBridge/Haswell counters in PCM. 
Secure Boot detected. Using Linux perf for uncore PMU programming.

Detected Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz "Intel(r) microarchitecture codename Kabylake" stepping 12
Unsupported processor model (158).
For processor-level memory bandwidth statistics please use pcm.x
Cleaning up
 Zeroed uncore PMU registers

There still seems to be a problem with reading the counters. I ran it as root.

Best,
Erik

@opcm
Copy link

opcm commented Oct 2, 2020

@ErikP94 you should try pcm.x instead as the pcm-memory.x output suggested

@opcm
Copy link

opcm commented Oct 2, 2020 •
edited

opening /dev/mem failed: errno is 1 (Operation not permitted)

to workaround this, boot linux with iomem=relaxed option

@TomTheBear
Copy link
Member

Thanks @opcm for stepping in.

@ErikP94: Were you able to test pcm.x and/or the workaround?

@ErikP94
Copy link
Author

ErikP94 commented Oct 5, 2020

Thank you from me as well and sorry for the delay of my answer.

Yes, I can run pcm.x and I get an output looking like this:

EXEC  : instructions per nominal CPU cycle
 IPC   : instructions per CPU cycle
 FREQ  : relation to nominal CPU frequency='unhalted clock ticks'/'invariant timer ticks' (includes Intel Turbo Boost)
 AFREQ : relation to nominal CPU frequency while in active state (not in power-saving C state)='unhalted clock ticks'/'invariant timer ticks while in C0-state'  (includes Intel Turbo Boost)
 L3MISS: L3 (read) cache misses 
 L2MISS: L2 (read) cache misses (including other core's L2 cache *hits*) 
 L3HIT : L3 (read) cache hit ratio (0.00-1.00)
 L2HIT : L2 cache hit ratio (0.00-1.00)
 L3MPI : number of L3 (read) cache misses per instruction
 L2MPI : number of L2 (read) cache misses per instruction
 READ  : bytes read from main memory controller (in GBytes)
 WRITE : bytes written to main memory controller (in GBytes)
 IO    : bytes read/written due to IO requests to memory controller (in GBytes); this may be an over estimate due to same-cache-line partial requests
 TEMP  : Temperature reading in 1 degree Celsius relative to the TjMax temperature (thermal headroom): 0 corresponds to the max temperature
 energy: Energy in Joules


 Core (SKT) | EXEC | IPC  | FREQ  | AFREQ | L3MISS | L2MISS | L3HIT | L2HIT | L3MPI | L2MPI |  TEMP

   0    0     0.05   1.33   0.04    1.10      98 K    205 K    0.46    0.58    0.00    0.00     56
   1    0     0.03   1.05   0.03    1.13      75 K    149 K    0.46    0.55    0.00    0.00     55
   2    0     0.03   0.81   0.03    0.82      72 K    283 K    0.73    0.59    0.00    0.00     53
   3    0     0.15   1.20   0.12    0.98     589 K   1358 K    0.56    0.60    0.00    0.00     54
   4    0     0.10   1.08   0.09    0.97     368 K    874 K    0.57    0.48    0.00    0.00     55
   5    0     0.03   0.40   0.07    1.26      76 K    164 K    0.50    0.53    0.00    0.00     55
   6    0     0.02   0.80   0.02    1.02      74 K    191 K    0.58    0.52    0.00    0.00     53
   7    0     0.05   0.92   0.05    1.17     104 K    228 K    0.48    0.63    0.00    0.00     54
---------------------------------------------------------------------------------------------------------------
 SKT    0     0.06   0.98   0.06    1.03    1461 K   3455 K    0.56    0.56    0.00    0.00     53
---------------------------------------------------------------------------------------------------------------
 TOTAL  *     0.06   0.98   0.06    1.03    1461 K   3455 K    0.56    0.56    0.00    0.00     N/A

 Instructions retired: 1041 M ; Active cycles: 1063 M ; Time (TSC): 2306 Mticks ; C0 (active,non-halted) core residency: 5.57 %

 C1 core residency: 15.00 %; C3 core residency: 0.05 %; C6 core residency: 2.40 %; C7 core residency: 76.98 %;
 C0 package residency: 63.58 %; C2 package residency: 3.18 %; C3 package residency: 0.00 %; C6 package residency: 0.44 %; C7 package residency: 0.00 %; C8 package residency: 32.80 %; C9 package residency: 0.00 %; C10 package residency: 0.00 %;
                             ┌────────────────────────────────────────────────────────────────────────────────┐
 Core    C-state distribution│00001111111111116677777777777777777777777777777777777777777777777777777777777777│
                             └────────────────────────────────────────────────────────────────────────────────┘
                             ┌────────────────────────────────────────────────────────────────────────────────┐
 Package C-state distribution│00000000000000000000000000000000000000000000000000022288888888888888888888888888│
                             └────────────────────────────────────────────────────────────────────────────────┘

 PHYSICAL CORE IPC                 : 1.96 => corresponds to 48.96 % utilization for cores in active state
 Instructions per nominal CPU cycle: 0.11 => corresponds to 2.82 % core utilization over time interval
 SMI count: 0
---------------------------------------------------------------------------------------------------------------
MEM (GB)->|  READ |  WRITE |   IO   | CPU energy |
---------------------------------------------------------------------------------------------------------------
 SKT   0     0.00     0.00     0.00       2.61              
---------------------------------------------------------------------------------------------------------------

It still doesn't show me infos of the memory controller, but maybe I'm missing out something huge here. I'm sorry, but I quite honestly left my known territory here. I will try the workaround now.

@opcm
Copy link

opcm commented Oct 5, 2020

@ErikP94 yes, please try the workaround for /mem/dev . Also posting the entire output from pcm is more helpful (they contain warnings and errors during initialization).

@ErikP94
Copy link
Author

ErikP94 commented Oct 5, 2020

I changed GRUB_CMDLINE_LINUX_DEFAULT="iomem=relaxed quiet splash" in the /etc/default/grub, ran update grub, rebooted and ran modprobe msr again.

It still tells me: opening /dev/mem failed: errno is 1 (Operation not permitted).

This is the begin of the output of pcm.x. I ran everything from root.

Processor Counter Monitor  ($Format:%ci ID=%h$)


IBRS and IBPB supported  : yes


 Processor Counter Monitor  ($Format:%ci ID=%h$)


IBRS and IBPB supported  : yes
STIBP supported          : yes
Spec arch caps supported : yes
Number of physical cores: 4
Number of logical cores: 8
Number of online logical cores: 8
Threads (logical cores) per physical core: 2
Num sockets: 1
Physical cores per socket: 4
Core PMU (perfmon) version: 4
Number of core PMU generic (programmable) counters: 4
Width of generic (programmable) counters: 48 bits
Number of core PMU fixed counters: 3
Width of fixed counters: 48 bits
Nominal core frequency: 2300000000 Hz
IBRS enabled in the kernel   : yes
STIBP enabled in the kernel  : no
The processor is not susceptible to Rogue Data Cache Load: yes
The processor supports enhanced IBRS                     : yes
Package thermal spec power: 15 Watt; Package minimum power: 0 Watt; Package maximum power: 0 Watt;
opening /dev/mem failed: errno is 1 (Operation not permitted)
Can not read memory controller counter information from PCI configuration space. Access to memory bandwidth counters is not possible.
You must be root to access these SandyBridge/IvyBridge/Haswell counters in PCM. 
Secure Boot detected. Using Linux perf for uncore PMU programming.
Trying to use Linux perf events...
Successfully programmed on-core PMU using Linux perf

Detected Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz "Intel(r) microarchitecture codename Kabylake" stepping 12

Best,
Erik

@opcm
Copy link

opcm commented Oct 5, 2020

ok. Thanks for testing. Linux forbids access to /dev/mem because the system is booted in the "Secure Boot" mode. There should be an option in the BIOS to disable it.

@ErikP94
Copy link
Author

ErikP94 commented Oct 5, 2020

Thanks for the hint!
I get the counters, when I disable secure boot!

@opcm
Copy link

opcm commented Oct 5, 2020

@ErikP94 thanks a lot for the confirmation!

@TomTheBear
Copy link
Member

Thanks for the confirmation. I documented the case in the FAQ.

@bhavanishankar25
Copy link

likwid-perfctr -C S0:1 -g BRANCH ./word

CPU name: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
CPU type: Intel Coffeelake processor
CPU clock: 2.81 GHz
ERROR - [./src/includes/perfmon_skylake.h:perfmon_init_skylake:72] No such file or directory.
Cannot zero 0x3F1 (0x3F1)
ERROR - [./src/includes/perfmon_skylake.h:perfmon_init_skylake:77] No such file or directory.
Cannot zero 0x3F7 (0x3F7)
ERROR: No event in given event string can be configured.
Either the events or counters do not exist for the
current architecture. If event options are set, they might
be invalid.

Please help me!!!!!,y is it showing like this.

@bhavanishankar25
Copy link

Hello,

I'm trying to get an overview of arithmetic and main memory performance of my program using the MEM_DP group in likwid-perfctr.
Unfortunately all the memory counters just show 0:

--------------------------------------------------------------------------------
CPU name:	Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
CPU type:	Intel Kabylake processor
CPU clock:	2.30 GHz
--------------------------------------------------------------------------------
 FLOPS/s =    243.49446939835948      MFLOPS/s
 FLOPS =    771147000.00000000      nflops
 Time =    3.1670000553131104      s
 Datasize =    158.18400000000000       MB
 Loads =    1542294000.0000000     
--------------------------------------------------------------------------------
Region likwid_compute, Group 1: MEM_DP
+-------------------+----------+
|    Region Info    |  Core 0  |
+-------------------+----------+
| RDTSC Runtime [s] | 3.346748 |
|     call count    |        1 |
+-------------------+----------+

+------------------------------------------+---------+-------------+
|                   Event                  | Counter |    Core 0   |
+------------------------------------------+---------+-------------+
|             INSTR_RETIRED_ANY            |  FIXC0  | 48479620000 |
|           CPU_CLK_UNHALTED_CORE          |  FIXC1  | 13018140000 |
|           CPU_CLK_UNHALTED_REF           |  FIXC2  |  7591400000 |
|              PWR_PKG_ENERGY              |   PWR0  |     52.1849 |
|              PWR_DRAM_ENERGY             |   PWR3  |      3.2621 |
| FP_ARITH_INST_RETIRED_128B_PACKED_DOUBLE |   PMC0  |           0 |
|    FP_ARITH_INST_RETIRED_SCALAR_DOUBLE   |   PMC1  |   800809100 |
| FP_ARITH_INST_RETIRED_256B_PACKED_DOUBLE |   PMC2  |           0 |
|                DRAM_READS                | MBOX0C1 |      -      |
|                DRAM_WRITES               | MBOX0C2 |      -      |
+------------------------------------------+---------+-------------+

+-----------------------------------+-----------+
|               Metric              |   Core 0  |
+-----------------------------------+-----------+
|        Runtime (RDTSC) [s]        |    3.3467 |
|        Runtime unhalted [s]       |    5.6503 |
|            Clock [MHz]            | 3950.9693 |
|                CPI                |    0.2685 |
|             Energy [J]            |   52.1849 |
|             Power [W]             |   15.5927 |
|          Energy DRAM [J]          |    3.2621 |
|           Power DRAM [W]          |    0.9747 |
|            DP [MFLOP/s]           |  239.2798 |
|          AVX DP [MFLOP/s]         |         0 |
|          Packed [MUOPS/s]         |         0 |
|          Scalar [MUOPS/s]         |  239.2798 |
|  Memory load bandwidth [MBytes/s] |         0 |
|  Memory load data volume [GBytes] |         0 |
| Memory evict bandwidth [MBytes/s] |         0 |
| Memory evict data volume [GBytes] |         0 |
|    Memory bandwidth [MBytes/s]    |         0 |
|    Memory data volume [GBytes]    |         0 |
|       Operational intensity       |    inf    |
+-----------------------------------+-----------+

I get the same results with and without the Marker API.
My testprogram is written in Fortran, I used the
gcc (Ubuntu 9.3.0-10ubuntu2) 9.3.0
to compile likwid.

The Fortran Module is compiled with gfortran (same version as gcc naturally). The changes in the config.mk are the following:

COMPILER = GCC#NO SPACE

# Path were to install likwid
PREFIX ?= /usr/local/likwid#NO SPACE

ACCESSMODE = perf_event#NO SPACE

FORTRAN_INTERFACE = true#NO SPACE

The changes in the make/inclue_GCC.mk are the following:

FC  = gfortran

FCFLAGS  = -fmodules ./  # gfortran

The command I run is:

sudo likwid-perfctr -C 0 -g MEM_DP ./likwid_testprogram

I hope I haven't overlooked anything and I'm sorry if I did!
Hope you can help me out, thanks in advance!

Erik

likwid-perfctr -C S0:1 -g BRANCH ./word

CPU name: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
CPU type: Intel Coffeelake processor
CPU clock: 2.81 GHz
ERROR - [./src/includes/perfmon_skylake.h:perfmon_init_skylake:72] No such file or directory.
Cannot zero 0x3F1 (0x3F1)
ERROR - [./src/includes/perfmon_skylake.h:perfmon_init_skylake:77] No such file or directory.
Cannot zero 0x3F7 (0x3F7)
ERROR: No event in given event string can be configured.
Either the events or counters do not exist for the
current architecture. If event options are set, they might
be invalid.

Please help me!!!!!,y is it showing like this.

@TomTheBear
Copy link
Member

Please open a separate issue. It's a different issue and does not fit this one.

@bhavanishankar25
Copy link

bhavanishankar25 commented Nov 17, 2020 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@TomTheBear @opcm @bhavanishankar25 @ErikP94