rocprof error hipRegisterApiCallback(0)

[psychocoderHPC]

I am working on porting PIConGPU to rocm and tried to trace my application with rocprof

I am working with rocm/2.10 and tried rocprof with a mini app without issues. When I try to trace PIConGPU I got an error I do not know how I can solve it.

Can someone please explain what this error means and how I can solve it?

mpiexec -n 1  rocprof --hip-trace  --timestamp on  ./bin/picongpu -d 1 1 1 -g  128 128 128 -s 100 --periodic 1 0 0 
RPL: on '191213_070919' from '/opt/rocm/rocprofiler' in 'workspace/buildPIC/khi'
RPL: profiling '"./bin/picongpu" "-d" "1" "1" "1" "-g" "128" "128" "128" "-s" "100" "--periodic" "1" "0" "0"'
RPL: input file ''
RPL: output dir '/tmp/rpl_data_191213_070919_42785'
RPL: result dir '/tmp/rpl_data_191213_070919_42785/input_results_191213_070919'
HIP_DB=0x1 [api]
ROCTracer (pid=42804): 
    HIP-trace()
roctracer_enable_domain_callback(), roctracer_enable_callback_fun(), HIP error: hipRegisterApiCallback(0) error(1011)



RPL: 'results.csv' is generated

rocm is installed in /shared/opt/rocm/2.10/ and in /opt/rocm is a older version of rocm. The error message shows RPL: on '191213_070919' from '/opt/rocm/rocprofiler' in 'workspace/buildPIC/khi'
Could it be that I need to set some environment variables that rocprof is not looking into the default install path?

[psychocoderHPC]

Found why /opt/rocm/rocprofiler is shown. rocprof -> /opt/rocm/rocprofiler/bin/rpl_run.sh is only a link to /opt/rocm...
Never the less the error message saying nothing to me :-(

[psychocoderHPC]

hip error 1011 means that a nullptr is passsed to an api function: hipRegisterApiCallback(0)
Now I understand what the error is saying but I still not know where this call is coming from. It must be something within rocprof.
Is it possible for a user to trace the error e.g. enable debug logs?

[eshcherb]

Do you use the standard HIP version from 2.10 release?
Could you send me output of the following command:
$ hipcc --version

You can use 'rocprof' with absolute path '/shared/opt/rocm/2.10/bin/rocprof'.

The 1011 HIP error is from 2.10 release:
hipErrorInvalidValue = 1011 (https://github.com/ROCm-Developer-Tools/HIP/blob/roc-2.10.0/include/hip/hip_runtime_api.h#L234)

And actually the error tells that on an attempt to register a callback for op-id '0', HIP returned error 'hipErrorInvalidValue' which looks quite strange which usually happens when HIP is built without tracing support.
Registering of NULL callback would cause the callback disabling.

[psychocoderHPC]

@eshcherb thanks for the explanation of error.

The hip version is

hipcc --version
HIP version: 2.10.19436-815a601
clang version 10.0.0 
Target: x86_64-unknown-linux-gnu
Thread model: posix

I will ask the guy who compiled the HIP environment for me. The strange thing is that it is working for me on the same system for a small example https://github.com/ComputationalRadiationPhysics/cupla/tree/dev/example/CUDASamples/matrixMul

I will provide today the steps to reproduce it and the system information via mail to Malaya, Nicholas from AMD via mail. I think the information will than somehow be push internally to you.
Since I work on an CRAY system where I can not publish all information to the public I hope that I can provide here the solution after the issue is solved so that other developers in the community can find it here.

[eshcherb]

Hi @psychocoderHPC , do I understand right that you have standard ROCm setup in '/opt/rocm' and non-standard in '/shared/opt/rocm/2.10'?

'rocprof' by default sets HIP_PATH to '/opt/rocm/hip' and the standard HIP version from ROCm release should have tracing enabled.
Could you try to set env HIP_PATH to custom location and to rerun your small example?
export HIP_PATH=/shared/opt/rocm/2.10/hip

To build HIP with enabled tracing you should use the following CMAKE parameters:
-DUSE_PROF_API=1 -DPROF_API_HEADER_PATH=<ROCm path>/roctracer/include/ext
And please check the CMAKE messages, it should print:
"Profiling API: <path to found prof_protocol.h header>"

[eshcherb]

Also from your previous message:
rocprof -> /opt/rocm/rocprofiler/bin/rpl_run.sh is only a link to /opt/rocm...

Actually '<ROCm path>/bin/rocprof' should be a relative link:
/opt/rocm/bin/rocprof -> ../rocprofiler/bin/rpl_run.sh
So profiler will be used from the same ROCm setup the 'rocprof' was called from.

I will fix the script to not set HIP_PATH and will check what is ROCm policy for choosing ROCm setup because the issue seems be related for two ROCm setups interference.

BTW you can use 'strace' to check which exactly HIP library was used:
$ rocprof strace <cmd-line>

[eshcherb]

Another way is to use LD_DEBUG env var.
I would appreciate if you can help to resolve the confusion with different behavior of your two tests.
Could you use LD_DEBUG env var to check which actual HIP library 'libhip_hcc.so' is used?:
$ LD_DEBUG=libs rocprof <cmdline> 2>&1 | grep hip_hcc

[psychocoderHPC]

matrix multiplication (profiling is working)

export LD_DEBUG=libs
mpiexec -n 1  /opt/rocm/bin/rocprof --hip-trace  --timestamp on ./matrixMul 2>&1 | grep hip_hcc

...
     20358:	find library=libhip_hcc.so [0]; searching
     20358:	  trying file=/opt/rocm/hcc/bin/../lib/libhip_hcc.so
     20358:	  trying file=/opt/rocm//hip/lib/libhip_hcc.so
...
     20358:	 search path=tls/x86_64:tls:x86_64:		(RPATH from file /opt/rocm//hip/lib/libhip_hcc.so)
     20358:	calling init: /opt/rocm//hip/lib/libhip_hcc.so
     20358:	calling fini: /opt/rocm//hip/lib/libhip_hcc.so [0]
...

PIConGPU (profiling is not working)

export LD_DEBUG=libs
/opt/rocm/bin/rocprof --hip-trace  --timestamp on  ./bin/picongpu -d 1 1 1 -g  128 128 128 -s 100 --periodic 0 0 0  2>&1 | grep hip_hcc

...
     38637:	find library=libhip_hcc.so [0]; searching
     38637:	  trying file=/usr/mpi/gcc/openmpi-4.0.0rc5/lib64/tls/x86_64/libhip_hcc.so
     38637:	  trying file=/usr/mpi/gcc/openmpi-4.0.0rc5/lib64/tls/libhip_hcc.so
     38637:	  trying file=/usr/mpi/gcc/openmpi-4.0.0rc5/lib64/x86_64/libhip_hcc.so
     38637:	  trying file=/usr/mpi/gcc/openmpi-4.0.0rc5/lib64/libhip_hcc.so
     38637:	  trying file=/opt/cray/pe/lib64/tls/x86_64/libhip_hcc.so
     38637:	  trying file=/opt/cray/pe/lib64/tls/libhip_hcc.so
     38637:	  trying file=/opt/cray/pe/lib64/x86_64/libhip_hcc.so
     38637:	  trying file=/opt/cray/pe/lib64/libhip_hcc.so
     38637:	  trying file=/opt/cray/pe/lib64/cce/tls/x86_64/libhip_hcc.so
     38637:	  trying file=/opt/cray/pe/lib64/cce/tls/libhip_hcc.so
     38637:	  trying file=/opt/cray/pe/lib64/cce/x86_64/libhip_hcc.so
     38637:	  trying file=/opt/cray/pe/lib64/cce/libhip_hcc.so
     38637:	  trying file=/opt/cray/lib64/tls/x86_64/libhip_hcc.so
     38637:	  trying file=/opt/cray/lib64/tls/libhip_hcc.so
     38637:	  trying file=/opt/cray/lib64/x86_64/libhip_hcc.so
     38637:	  trying file=/opt/cray/lib64/libhip_hcc.so
     38637:	  trying file=/home/users/XXX/opt/install/boost_1_71_0/lib/tls/x86_64/libhip_hcc.so
     38637:	  trying file=/home/users/XXX/opt/install/boost_1_71_0/lib/tls/libhip_hcc.so
     38637:	  trying file=/home/users/XXX/opt/install/boost_1_71_0/lib/x86_64/libhip_hcc.so
     38637:	  trying file=/home/users/XXX/opt/install/boost_1_71_0/lib/libhip_hcc.so
     38637:	  trying file=/usr/mpi/gcc/openmpi-4.0.0rc5/lib64/libhip_hcc.so
     38637:	  trying file=/home/users/XXX/c-hip-clang-build/HIP/install/lib/tls/x86_64/libhip_hcc.so
     38637:	  trying file=/home/users/XXX/c-hip-clang-build/HIP/install/lib/tls/libhip_hcc.so
     38637:	  trying file=/home/users/XXX/c-hip-clang-build/HIP/install/lib/x86_64/libhip_hcc.so
     38637:	  trying file=/home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so
     38637:	 search path=/opt/rocm/hcc/bin/../lib/tls/x86_64:/opt/rocm/hcc/bin/../lib/tls:/opt/rocm/hcc/bin/../lib/x86_64:/opt/rocm/hcc/bin/../lib		(RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
bin/../lib		(RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
     38637:	 search path=/opt/rocm/hcc/bin/../lib		(RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
     38637:	 search path=/opt/rocm/hcc/bin/../lib		(RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
     38637:	 search path=/opt/rocm/hcc/bin/../lib		(RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
     38637:	 search path=/opt/rocm/hcc/bin/../lib		(RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
     38637:	 search path=/opt/rocm/hcc/bin/../lib		(RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
     38637:	 search path=/opt/rocm/hcc/bin/../lib		(RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
     38637:	 search path=/opt/rocm/hcc/bin/../lib		(RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
     38637:	 search path=/opt/rocm/hcc/bin/../lib		(RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
     38637:	 search path=/opt/rocm/hcc/bin/../lib		(RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
     38637:	calling init: /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so
     38637:	 search path=/opt/rocm/hcc/bin/../lib		(RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
     38637:	 search path=/opt/rocm/hcc/bin/../lib		(RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)

[psychocoderHPC]

@eshcherb Thanks for your help. The problem was that libhip_hcc.so was used from /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so (thats the compiled HIP-clang location).

By setting LD_LIBRARY_PATH to /opt/rocm//hcc/lib:$LD_LIBRARY_PATH I was able get rid of the error hipRegisterApiCallback(0)

[psychocoderHPC]

To build HIP with enabled tracing you should use the following CMAKE parameters:
-DUSE_PROF_API=1 -DPROF_API_HEADER_PATH=/roctracer/include/ext

I am now able to profile my application.
This is very important, I missed it. I compiled now all by myself and added this flags.