Caliper integration with new PFA (Clang 18)

[tpatki]

Supersedes #130.

Todo:

• Port the Caliper integration to work with NewPassManager (C++ Integration)
• Test with clang18/amdclang18
• Add Python integration for Caliper (and document). Python bindings in caliper enabled here: Adds Python bindings to Caliper with pybind11 llnl/Caliper#573. Example use in hatchet here: Adds an annotation-based profiling API for measuring the performance of Hatchet and Thicket llnl/hatchet#142
• Test Python with and without caliper
• Update docs to include the caliper-enable option and an associated example

[tpatki]

Update 5/14/2025.

I ported the older Caliper integration over to Clang18 and resolved the various build issues (hopefully correctly). However, there is a strange issue that is showing up in my build in Tioga, that I am not sure how to resolve. I can build PFA but can't load the pass. It fails with an undefined symbol: _ZN4llvm24DisableABIBreakingChecksE error.

The surprising thing is that I don't run into this in the Github CI, things build correctly there. So looks like this is specific to our systems.

Seems unrelated to Caliper-related changes.

Looks like an ABI compatibility issue...

Not sure what to try next just yet...

$ make
[  4%] [BISON][parser] Building parser with bison 3.0.4
[  8%] [FLEX][lexer] Building scanner with flex 2.6.1
[ 12%] Building CXX object parser/CMakeFiles/perfflow_parser.dir/perfflow_parser.cpp.o
[ 16%] Building CXX object parser/CMakeFiles/perfflow_parser.dir/parser.tab.cpp.o
[ 20%] Building CXX object parser/CMakeFiles/perfflow_parser.dir/lex.yy.cpp.o
[ 24%] Linking CXX shared library libperfflow_parser.so
[ 24%] Built target perfflow_parser
[ 28%] Building CXX object runtime/CMakeFiles/perfflow_runtime.dir/advice_chrome_tracing.cpp.o
[ 32%] Building CXX object runtime/CMakeFiles/perfflow_runtime.dir/advice_dispatcher.cpp.o
[ 36%] Building CXX object runtime/CMakeFiles/perfflow_runtime.dir/perfflow_runtime.cpp.o
[ 40%] Linking CXX shared library libperfflow_runtime.so
[ 40%] Built target perfflow_runtime
[ 44%] Building CXX object weaver/weave/CMakeFiles/WeavePass.dir/perfflow_weave_common.cpp.o
[ 48%] Building CXX object weaver/weave/CMakeFiles/WeavePass.dir/perfflow_weave_new_pass.cpp.o
[ 52%] Linking CXX shared module libWeavePass.so
[ 52%] Built target WeavePass
[ 56%] Building CXX object test/CMakeFiles/smoketest.dir/smoketest.cpp.o
error: unable to load plugin '../weaver/weave/libWeavePass.so': '../weaver/weave/libWeavePass.so: undefined symbol: _ZN4llvm24DisableABIBreakingChecksE'
make[2]: *** [test/CMakeFiles/smoketest.dir/build.make:76: test/CMakeFiles/smoketest.dir/smoketest.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:399: test/CMakeFiles/smoketest.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

[slabasan]

The build works for me on tioga:

1. Build and install caliper using same compilers as you intend to build PFA+caliper:

cmake \
-DCMAKE_C_COMPILER=/opt/rocm-6.3.1/lib/llvm/bin/amdclang \
-DCMAKE_CXX_COMPILER=/opt/rocm-6.3.1/lib/llvm/bin/amdclang++ \
-DCMAKE_INSTALL_PREFIX=caliper-install \
..

2. Build PFA, link to caliper install:

cmake \
-C ../host-configs/tioga-4.18.0-x86_64-rocm@6.3.1-amdclang@18.0.0.cmake \
-Dcaliper_DIR=caliper-install \
..

And I can run one of the smoketests and get a pfw file and see caliper log outputs:

$ CALI_LOG_VERBOSITY=2 ./test/smoketest
Inside main
== CALIPER: Available services: aggregate,alloc,allocstats,async_event,cpuinfo,debug,event,io,loop_monitor,loop_statistics,memstat,pthread,recorder,region_monitor,report,statistics,sysalloc,timer,timeseries,timestamp,trace,validator
== CALIPER: Initialized
== CALIPER: No manual config specified, disabling default channel
foo
bar
bas
foo
bar
bas
foo
bar
bas
foo
bar
bas
== CALIPER: Finalizing ... 
== CALIPER: Releasing Caliper global data.
  Max active channels: 0
== CALIPER: Process blackboard: max 1 entries (0.0979432% occupancy).
== CALIPER: Releasing Caliper thread data: 
  Metadata tree: 1 blocks, 31 nodes
   Metadata memory pool: 1 MiB reserved, 18.2275 KiB used
  Thread blackboard: max 1 entries (0.0979432% occupancy).
== CALIPER: Finished

[slabasan]

Some notes about adding python interface:

from pycaliper import annotate_function
from pycaliper.instrumentation import begin_region, end_region

Add begin_region(name) and end_region(name) around regions of interest. Ex:

begin_region("get_perf_data_for_node")
if isinstance(dframe.index, pd.MultiIndex):
    row = pd.concat([dframe.loc[node]], keys=[node], names=["node"])
else:
    row = dframe.loc[node]
end_region("get_perf_data_for_node")

Or if wanting to go the decorator route:

@annotate_function()
def __init__(self, attrs=None, **kwargs):
    """Construct a frame from a dictionary, or from immediate kwargs.

[tpatki]

Update 5/20/2025:

Reporting success with the PFA+Caliper Python interface integration as well. Tests are passing.
To do: update docs and add a python example.

$ PERFFLOW_OPTIONS="caliper-enable=True" CALI_CONFIG=runtime-report ./smoketest.py 

Inside main
foo
bar
bas
foo
bar
bas
foo
bar
bas
foo
bar
bas
Path                                 Time (E) Time (I) Time % (E) Time % (I) 
<function foo at 0x155546a24540>     0.004692 0.009423  47.547246  95.486397 
  <function bar at 0x155546a24400>   0.004659 0.004731  47.213609  47.939151 
    <function bas at 0x15554732de40> 0.000072 0.000072   0.725542   0.725542 

[tpatki]

@slabasan
This is ready to squash & merge once tests pass.
Please review, just in case I missed something.