Add PAPI 7.0.1 and 7.1.0 #38443

wyphan · 2023-06-16T15:06:51Z

Supersedes #33940

SHA256 sum computed manually after verifying the MD5 sum with the one posted at the ICL website.

var/spack/repos/builtin/packages/papi/package.py

wyphan · 2023-06-16T16:35:09Z

Confirmed working on my dev rig (AMD Zen 2, NVIDIA "Ampere" 8.6, AMDgfx900, Ubuntu 20.04.6):

wyp@basecamp:~/work/spack$ spack find
==> In environment test
==> Root specs
papi@7.0.1 ~cuda~nvml~rocm~rocm_smi  papi@7.0.1 +cuda+nvml~rocm~rocm_smi
papi@7.0.1 ~cuda~nvml+rocm+rocm_smi  papi@7.0.1 +cuda+nvml+rocm+rocm_smi

wyp@basecamp:~/work/spack$ spack install
:
[ install succeeded ]

wyp@basecamp:~/work/spack$ spack load papi +cuda +rocm

wyp@basecamp:~/work/spack$ papi_avail -a
Available PAPI preset and user defined events plus hardware information.
--------------------------------------------------------------------------------
PAPI version             : 7.0.1.0
Operating system         : Linux 5.15.0-75-generic
Vendor string and code   : AuthenticAMD (2, 0x2)
Model string and code    : AMD Ryzen 5 3600X 6-Core Processor (113, 0x71)
CPU revision             : 0.000000
CPUID                    : Family/Model/Stepping 23/113/0, 0x17/0x71/0x00
CPU Max MHz              : 4000
CPU Min MHz              : 2200
Total cores              : 12
SMT threads per core     : 2
Cores per socket         : 6
Sockets                  : 1
Cores per NUMA region    : 12
NUMA regions             : 1
Running in a VM          : no
Number Hardware Counters : 5
Max Multiplex Counters   : 384
Fast counter read (rdpmc): yes
--------------------------------------------------------------------------------

================================================================================
  PAPI Preset Events
================================================================================
    Name        Code    Deriv Description (Note)

PAPI_L1_ICM  0x80000001  No   Level 1 instruction cache misses
PAPI_L2_DCM  0x80000002  No   Level 2 data cache misses
PAPI_L2_ICM  0x80000003  No   Level 2 instruction cache misses
PAPI_TLB_DM  0x80000014  No   Data translation lookaside buffer misses
PAPI_TLB_IM  0x80000015  Yes  Instruction translation lookaside buffer misses
PAPI_BR_TKN  0x8000002c  No   Conditional branch instructions taken
PAPI_BR_MSP  0x8000002e  No   Conditional branch instructions mispredicted
PAPI_TOT_INS 0x80000032  No   Instructions completed
PAPI_FP_INS  0x80000034  No   Floating point instructions
PAPI_BR_INS  0x80000037  No   Branch instructions
PAPI_TOT_CYC 0x8000003b  No   Total cycles
PAPI_L2_DCH  0x8000003f  No   Level 2 data cache hits
PAPI_L1_DCA  0x80000040  No   Level 1 data cache accesses
PAPI_L2_DCR  0x80000044  No   Level 2 data cache reads
PAPI_L2_ICH  0x8000004a  No   Level 2 instruction cache hits
PAPI_L2_ICR  0x80000050  No   Level 2 instruction cache reads
PAPI_FP_OPS  0x80000066  No   Floating point operations
--------------------------------------------------------------------------------
Of 17 available events, 1 is derived.

wyp@basecamp:~/work/spack$ export PAPI_ROCMSMI_ROOT=/opt/rocm-5.4.3/rocm_smi

wyp@basecamp:~/work/spack$ papi_component_avail 
Available components and hardware information.
--------------------------------------------------------------------------------
PAPI version             : 7.0.1.0
Operating system         : Linux 5.15.0-75-generic
Vendor string and code   : AuthenticAMD (2, 0x2)
Model string and code    : AMD Ryzen 5 3600X 6-Core Processor (113, 0x71)
CPU revision             : 0.000000
CPUID                    : Family/Model/Stepping 23/113/0, 0x17/0x71/0x00
CPU Max MHz              : 4000
CPU Min MHz              : 2200
Total cores              : 12
SMT threads per core     : 2
Cores per socket         : 6
Sockets                  : 1
Cores per NUMA region    : 12
NUMA regions             : 1
Running in a VM          : no
Number Hardware Counters : 5
Max Multiplex Counters   : 384
Fast counter read (rdpmc): yes
--------------------------------------------------------------------------------

Compiled-in components:
Name:   perf_event              Linux perf_event CPU counters
Name:   perf_event_uncore       Linux perf_event CPU uncore and northbridge
   \-> Disabled: Insufficient permissions for uncore access.  Set /proc/sys/kernel/perf_event_paranoid to 0 or run as root.
Name:   example                 A simple example component
Name:   cuda                    CUDA events and metrics via NVIDIA CuPTI interfaces
Name:   nvml                    NVML provides the API for monitoring NVIDIA hardware (power usage, temperature, fan speed, etc)
Name:   rocm                    GPU events and metrics via AMD ROCm-PL API
Name:   rocm_smi                AMD GPU System Management Interface via rocm_smi_lib
   \-> Disabled: Error while initializing the native event table.
Name:   sysdetect               System info detection component

Active components:
Name:   perf_event              Linux perf_event CPU counters
                                Native: 141, Preset: 17, Counters: 5
                                PMUs supported: perf, perf_raw, amd64_fam17h_zen2

Name:   example                 A simple example component
                                Native: 4, Preset: 0, Counters: 3

Name:   cuda                    CUDA events and metrics via NVIDIA CuPTI interfaces
                                Native: 259254, Preset: 0, Counters: 259254

Name:   nvml                    NVML provides the API for monitoring NVIDIA hardware (power usage, temperature, fan speed, etc)
                                Native: 28, Preset: 0, Counters: 28

Name:   rocm                    GPU events and metrics via AMD ROCm-PL API
                                Native: 0, Preset: 0, Counters: 0

Name:   sysdetect               System info detection component
                                Native: 0, Preset: 0, Counters: 0
--------------------------------------------------------------------------------

naromero77

Thanks!

naromero77 · 2023-06-17T13:52:12Z

There is a failure that seems to be unrelated to this patch, going to restart the pipeline.

naromero77 · 2023-06-17T13:52:24Z

@spackbot run pipeline

spackbot-app · 2023-06-17T13:52:30Z

I've started that pipeline for you!

eugeneswalker · 2023-06-18T17:16:06Z

(specs) papi/jjh2oak 7.0.1 cce@=15.0.1 linux-rhel8-zen4 E4S Cray

...
ftn-78 ftn: ERROR in command line
  The -f option has an invalid argument, "ree".
make[2]: *** [Makefile:46: query_device_simple_f] Error 1
make[2]: Leaving directory '/tmp/gitlab-runner-1/spack-stage/spack-stage-papi-7.0.1-jjh2oakmkillg524lzt2eo2heg6yqet6/spack-src/src/components/sysdetect/tests'
make[1]: *** [Makefile.inc:265: comp_tests] Error 2
make[1]: Leaving directory '/tmp/gitlab-runner-1/spack-stage/spack-stage-papi-7.0.1-jjh2oakmkillg524lzt2eo2heg6yqet6/spack-src/src'
make: *** [Rules.pfm4_pe:45: libpfm4/lib/libpfm.a] Error 2
==> Error: ProcessError: Command exited with status 2:
    'make' '-j1' 'V=1'
3 errors found in build log:
     1334    make[2]: Entering directory '/tmp/gitlab-runner-1/spack-stage/spac
             k-stage-papi-7.0.1-jjh2oakmkillg524lzt2eo2heg6yqet6/spack-src/src/
             components/sysdetect/tests'
     1335    /home/gitlab-runner-1/builds/Q2MuKA89/0/spack/spack/lib/spack/env/
             cce/cc -g  -DPAPI_NUM_COMP=4 -O2 -I. -I../../.. -I../../../testlib
              -I../../../validation_tests -I/home/gitlab-runner-1/builds/Q2MuKA
             89/0/spack/spack/opt/spack/__spack_path_placeholder__/__spack_path
             _placeholder__/__spack_path_placeholder__/__spack_path_placeholder
             __/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
             ath_placeholder__/__spa/morepadding/linux-rhel8-zen4/cce-15.0.1/pa
             pi-7.0.1-jjh2oakmkillg524lzt2eo2heg6yqet6/include -c -o query_devi
             ce_simple.o query_device_simple.c
     1336    /home/gitlab-runner-1/builds/Q2MuKA89/0/spack/spack/lib/spack/env/
             cce/cc -g  -DPAPI_NUM_COMP=4 -I. -I../../.. -I../../../testlib -I.
             ./../../validation_tests -I/home/gitlab-runner-1/builds/Q2MuKA89/0
             /spack/spack/opt/spack/__spack_path_placeholder__/__spack_path_pla
             ceholder__/__spack_path_placeholder__/__spack_path_placeholder__/_
             _spack_path_placeholder__/__spack_path_placeholder__/__spack_path_
             placeholder__/__spa/morepadding/linux-rhel8-zen4/cce-15.0.1/papi-7
             .0.1-jjh2oakmkillg524lzt2eo2heg6yqet6/include -o query_device_simp
             le query_device_simple.o ../../../testlib/libtestlib.a ../../../li
             bpapi.a
     1337    /home/gitlab-runner-1/builds/Q2MuKA89/0/spack/spack/lib/spack/env/
             cce/ftn -free -I../../.. -o query_device_simple_f query_device_sim
             ple_f.F ../../../libpapi.a
     1338    ftn-78 ftn: ERROR in command line
     1339      The -f option has an invalid argument, "ree".
  >> 1340    make[2]: *** [Makefile:46: query_device_simple_f] Error 1
     1341    make[2]: Leaving directory '/tmp/gitlab-runner-1/spack-stage/spack
             -stage-papi-7.0.1-jjh2oakmkillg524lzt2eo2heg6yqet6/spack-src/src/c
             omponents/sysdetect/tests'
  >> 1342    make[1]: *** [Makefile.inc:265: comp_tests] Error 2
     1343    make[1]: Leaving directory '/tmp/gitlab-runner-1/spack-stage/spack
             -stage-papi-7.0.1-jjh2oakmkillg524lzt2eo2heg6yqet6/spack-src/src'
  >> 1344    make: *** [Rules.pfm4_pe:45: libpfm4/lib/libpfm.a] Error 2

G-Ragghianti · 2023-06-19T09:31:55Z

I have posted an issue on this to papi upstream: icl-utk-edu/papi#24

Do we want to maybe include a patch changing FFLAGS to something reasonable for CCE/FTN to get this going within spack?

wyphan · 2023-07-06T15:28:21Z

@srekolam @renjithravindrankannath if you inspect the CI failures above, it's a concretization error. I tried to reproduce it using spack ci reproduce-build, turns out it involves hip@master combined with rocprofiler-dev@4.5.2 and so on... I think this needs to be fixed at the ROCmPackage build system level instead of the llvm-amdgpu package level

naromero77 · 2023-12-21T17:50:58Z

@wyphan The upstream only has PAPI 6.x which is almost three years old. I guess there were issues with the LLVM-AMDGPU package that held up this PR. One possibility is to just include PAPI 7.0.1 and 7.1.0 in a single PR and fix the LLVM-AMDGPU in a different set of PRs.

wyphan · 2023-12-21T17:54:59Z

@naromero77 Yeah that sounds reasonable. I'll try to finalize the PR based on this plan later today.

naromero77

Thanks!

wyphan · 2023-12-21T19:53:09Z

Ugh. Looks like the SHA sum for the tarball from ICL website and the one in GitHub release tag are different. Fixed now to use the ICL website's tarball.

naromero77

LGTM.

I will merge tomorrow if I don't hear back from @G-Ragghianti or the others.

haampie · 2023-12-23T09:43:11Z

Hi @wyphan, Gitlab CI didn't report this issue with the PR: https://gitlab.spack.io/spack/spack/-/jobs/9653923

/home/gitlab-runner-protected-3/builds/Cj9btwqn/0/spack/spack/lib/spack/env/cce/ftn -ffree-form -ffree -I../../.. -o query_device_simple_f query_device_simple_f.F ../../../libpapi.a
ftn-78 ftn: ERROR in command line
The -f option has an invalid argument, "free-form".

Do you wanna take a look at that?

haampie · 2023-12-23T09:48:40Z

Notice you don't have to create patch files, you can just set make VARIABLE=VALUE

wyphan · 2023-12-23T13:56:41Z

Huh, I thought that CCE patch already got merged into the main codebase before the 7.1.0 release, so I only applied it for 7.0.1 in the recipe.

* Add PAPI 7.0.1 * Add comment about skipping PAPI 7.0.0 * Add patch to avoid adding Intel ifort/ifx flag on Cray ftn * Modify patch to include Cray-specific flags * Adjust recipe to always apply patch for 7.0.1 * Expand Cray compiler checks in patch * Forgot to update recipe * Adjust recipe so it looks for hipcc in the correct path * Revert "Adjust recipe so it looks for hipcc in the correct path" This reverts commit 0db3df4. * Patch HIP_PATH to work with Spack-built HIP * Patch LDFLAGS with llvm-amdgpu path * Forgot the depends_on line * libomptarget only builds with clang * Try a self-consistent build of llvm-amdgpu * Try making llvm-amdgpu depend on llvm for llvmoffloadarch library * Update prereq to use rocm-openmp-extras instead * Refactor llvm-amdgpu to use a version dict * Fix typo * Hack to exclude older versions without matching rocm-openmp-extras * Add PAPI 7.1.0 * Revert changes to llvm-amdgpu * Fix PAPI 7.1.0 checksum

Add PAPI 7.0.1

cb2c614

spackbot-app bot requested a review from G-Ragghianti June 16, 2023 15:06

spackbot-app bot added new-version update-package labels Jun 16, 2023

wyphan self-assigned this Jun 16, 2023

wyphan requested a review from naromero77 June 16, 2023 15:07

wyphan added update-package and removed update-package labels Jun 16, 2023

wyphan requested a review from alalazo June 16, 2023 15:08

naromero77 requested changes Jun 16, 2023

View reviewed changes

var/spack/repos/builtin/packages/papi/package.py Show resolved Hide resolved

Add comment about skipping PAPI 7.0.0

68383ce

wyphan requested a review from naromero77 June 16, 2023 15:42

naromero77 previously approved these changes Jun 16, 2023

View reviewed changes

alalazo assigned naromero77 and eugeneswalker Jun 19, 2023

G-Ragghianti mentioned this pull request Jun 19, 2023

Error building component tests for sysdetect,sde with Cray compiler icl-utk-edu/papi#24

Closed

Add patch to avoid adding Intel ifort/ifx flag on Cray ftn

6e1812a

wyphan dismissed naromero77’s stale review via 6e1812a June 19, 2023 18:55

spackbot-app bot added the patch label Jun 19, 2023

wyphan and others added 4 commits June 19, 2023 16:29

Modify patch to include Cray-specific flags

97ec957

Adjust recipe to always apply patch for 7.0.1

8327d9b

Expand Cray compiler checks in patch

15830ee

Forgot to update recipe

46bba64

gcongiu mentioned this pull request Jun 20, 2023

2023.06.20 fix fortran test flags icl-utk-edu/papi#25

Merged

3 tasks

wyphan added 4 commits June 26, 2023 16:30

Update prereq to use rocm-openmp-extras instead

b345361

Refactor llvm-amdgpu to use a version dict

9b9496e

Fix typo

752f6df

Hack to exclude older versions without matching rocm-openmp-extras

4a5ea3e

wyphan requested review from srekolam and renjithravindrankannath July 5, 2023 17:49

G-Ragghianti mentioned this pull request Oct 16, 2023

Installation issue: papi@master fails to build with %oneapi compiler #40398

Open

4 tasks

wyphan added 2 commits December 21, 2023 13:42

Add PAPI 7.1.0

17052f6

Revert changes to llvm-amdgpu

cfe6e5a

spackbot-app bot added resources virtual-dependencies labels Dec 21, 2023

wyphan changed the title ~~Add PAPI 7.0.1~~ Add PAPI 7.0.1 and 7.1.0 Dec 21, 2023

Merge branch 'develop' into wyphan/papi701

938a79a

wyphan requested review from naromero77 and eugeneswalker December 21, 2023 19:11

naromero77 previously approved these changes Dec 21, 2023

View reviewed changes

Fix PAPI 7.1.0 checksum

151e865

wyphan dismissed naromero77’s stale review via 151e865 December 21, 2023 19:52

naromero77 approved these changes Dec 21, 2023

View reviewed changes

naromero77 merged commit 327a7a4 into spack:develop Dec 22, 2023
13 checks passed

wyphan deleted the wyphan/papi701 branch December 22, 2023 21:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add PAPI 7.0.1 and 7.1.0 #38443

Add PAPI 7.0.1 and 7.1.0 #38443

wyphan commented Jun 16, 2023 •

edited

wyphan commented Jun 16, 2023 •

edited

naromero77 left a comment

naromero77 commented Jun 17, 2023

naromero77 commented Jun 17, 2023

spackbot-app bot commented Jun 17, 2023

eugeneswalker commented Jun 18, 2023

G-Ragghianti commented Jun 19, 2023

wyphan commented Jul 6, 2023

naromero77 commented Dec 21, 2023

wyphan commented Dec 21, 2023

naromero77 left a comment

wyphan commented Dec 21, 2023

naromero77 left a comment

haampie commented Dec 23, 2023

haampie commented Dec 23, 2023

wyphan commented Dec 23, 2023

Add PAPI 7.0.1 and 7.1.0 #38443

Add PAPI 7.0.1 and 7.1.0 #38443

Conversation

wyphan commented Jun 16, 2023 • edited

wyphan commented Jun 16, 2023 • edited

naromero77 left a comment

Choose a reason for hiding this comment

naromero77 commented Jun 17, 2023

naromero77 commented Jun 17, 2023

spackbot-app bot commented Jun 17, 2023

eugeneswalker commented Jun 18, 2023

G-Ragghianti commented Jun 19, 2023

wyphan commented Jul 6, 2023

naromero77 commented Dec 21, 2023

wyphan commented Dec 21, 2023

naromero77 left a comment

Choose a reason for hiding this comment

wyphan commented Dec 21, 2023

naromero77 left a comment

Choose a reason for hiding this comment

haampie commented Dec 23, 2023

haampie commented Dec 23, 2023

wyphan commented Dec 23, 2023

wyphan commented Jun 16, 2023 •

edited

wyphan commented Jun 16, 2023 •

edited