Error not progagated if jattach fails to load asyncProfiler lib

[vans239]

I built an async-profiler from sources (there was quite long period of nice features on master without releases)
When I try to use asprof it fails to execute action, but dodesn't show any error.

Example with status:

~ $ local-async-profiler/bin/asprof status 5385 
~ $ async-profiler-3.0-linux-x64/bin/asprof status 5385
Profiler is not active

It's much more confusing when you profile:

~ $ async-profiler-3.0-linux-x64/bin/asprof -d 10 5385
Profiling for 10 seconds
Done
--- Execution profile ---
Total samples       : 14
....
~ $  local-async-profiler/bin/asprof status -d 10 5385 
<<empty after 10 seconds>>

In my case root cause is

bin/../lib/libasyncProfiler.so was not loaded.
/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by local-async-profilerbin/../lib/libasyncProfiler.so)

[apangin]

It works as expected for me.
Please provide more details about your environment and how you build the project.

$ bin/asprof -d 5 jps
Target JVM failed to load /home/andrei/build/bin/../lib/libasyncProfiler.so
$ ldd lib/libasyncProfiler.so
lib/libasyncProfiler.so: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by lib/libasyncProfiler.so)
lib/libasyncProfiler.so: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by lib/libasyncProfiler.so)
lib/libasyncProfiler.so: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by lib/libasyncProfiler.so)
        linux-vdso.so.1 (0x00007ffe80e89000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f42f6848000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f42f6688000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f42f6505000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f42f6a2f000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f42f64eb000)

[vans239]

Build host (Fedora 28 ):

➜  async-profiler git:(master) echo "$(git show -s --format="%ci %H" HEAD)"
2024-01-28 18:06:55 +0000 03004b65058910c4e13f7e32482e01113fa7ab63
➜  async-profiler git:(master) make clean
rm -f -r build
➜  async-profiler git:(master) make  
mkdir -p build/bin
mkdir -p build/lib
for f in src/*.cpp; do echo '#include "'$f'"'; done |\
g++  -O3 -fno-exceptions -fno-omit-frame-pointer -fvisibility=hidden -Wl,-z,defs -fwhole-program -momit-leaf-frame-pointer -DPROFILER_VERSION=\"3.0\" -I/usr/java/jdk-21.0.1+12//include -Isrc/helper -I/usr/java/jdk-21.0.1+12//include/linux -fPIC -shared -o build/lib/libasyncProfiler.so -xc++ - -ldl -lpthread -lrt
cc  -O3 -fno-exceptions -DPROFILER_VERSION=\"3.0\" -o build/bin/asprof src/launcher/*.cpp src/jattach/*.c
strip build/bin/asprof
mkdir -p build/api
/usr/java/jdk-21.0.1+12//bin/javac --release 8 -Xlint:-options -d build/api src/api/one/profiler/Events.java src/api/one/profiler/AsyncProfiler.java src/api/one/profiler/AsyncProfilerMXBean.java src/api/one/profiler/Counter.java
/usr/java/jdk-21.0.1+12//bin/jar cf build/lib/async-profiler.jar -C build/api .
rm -f -r build/api
mkdir -p build/converter
/usr/java/jdk-21.0.1+12//bin/javac --release 8 -Xlint:-options -d build/converter src/converter/Classifier.java src/converter/jfr2flame.java src/converter/FlameGraph.java src/converter/CollapsedStacks.java src/converter/jfr2nflx.java src/converter/Main.java src/converter/Arguments.java src/converter/jfr2pprof.java src/converter/one/jfr/event/ObjectCount.java src/converter/one/jfr/event/ExecutionSample.java src/converter/one/jfr/event/EventAggregator.java src/converter/one/jfr/event/CPULoad.java src/converter/one/jfr/event/ContendedLock.java src/converter/one/jfr/event/Event.java src/converter/one/jfr/event/AllocationSample.java src/converter/one/jfr/event/LiveObject.java src/converter/one/jfr/event/GCHeapSummary.java src/converter/one/jfr/JfrClass.java src/converter/one/jfr/JfrReader.java src/converter/one/jfr/ClassRef.java src/converter/one/jfr/Dictionary.java src/converter/one/jfr/StackTrace.java src/converter/one/jfr/MethodRef.java src/converter/one/jfr/JfrField.java src/converter/one/jfr/Element.java src/converter/one/proto/Proto.java
/usr/java/jdk-21.0.1+12//bin/jar cfe build/lib/converter.jar Main -C build/converter . -C src/res .
rm -f -r build/converter

➜  async-profiler git:(master) uname -a
Linux local 4.16.5-300.fc28.x86_64 #1 SMP Fri Apr 27 17:38:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Remote host:

 $ strace -ff -v -s 1000  ./asprof status 53252
....
[pid 58101] socket(AF_UNIX, SOCK_STREAM, 0) = 3
[pid 58101] connect(3, {sa_family=AF_UNIX, sun_path="/proc/53252/root/tmp/.java_pid53252"}, 110) = 0
[pid 58101] write(3, "1\0load\0../build/bin/../lib/libasyncProfiler.so\0true\0status,file=/tmp/asprof.58100.53252,,log=/tmp/asprof-log.58100.53252\0", 137) = 137
[pid 58101] read(3, "0\n...build/bin/../lib/libasyncProfiler.so was not loaded.\n/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by .../build/bin/../lib/libasyncProfiler.so)\n", 8191) = 202
[pid 58101] read(3, "", 7989)           = 0
[pid 58101] close(3)                    = 0
[pid 58101] exit_group(0)               = ?
...
 $ uname -a 
Linux myhost.local 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

 $ ldd ../lib/libasyncProfiler.so
../lib/libasyncProfiler.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ../lib/libasyncProfiler.so)
../lib/libasyncProfiler.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by ../lib/libasyncProfiler.so)
../lib/libasyncProfiler.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ../lib/libasyncProfiler.so)
	linux-vdso.so.1 =>  (0x00007ffc3ebea000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007ff5d22e4000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff5d20c8000)
	librt.so.1 => /lib64/librt.so.1 (0x00007ff5d1ec0000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007ff5d1bb8000)
	libm.so.6 => /lib64/libm.so.6 (0x00007ff5d18b6000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff5d16a0000)
	libc.so.6 => /lib64/libc.so.6 (0x00007ff5d12d2000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ff5d2738000)

Is there anything else what I can provide?

[apangin]

Thanks. The most important piece here is the JVM version.
jattach/asprof correctly reported an error with JDK 8-20, but fails to do so with JDK 21.

Agent loading has been significantly refactored in JDK 21 as a result of JDK-8307478 and JDK-8257967. As a consequence of this refactoring, the protocol of reporting errors in Dynamic Attach has been changed, intentionally or not. I suspect this is a JVM bug, but need to investigate further.

// JDK 8
$ asprof jps load BAD_LIBRARY
Connected to remote JVM
JVM response code = -1

// JDK 11-20
$ asprof jps load BAD_LIBRARY
Connected to remote JVM
JVM response code = -1
BAD_LIBRARY was not loaded.
libBAD_LIBRARY.so: cannot open shared object file: No such file or directory

// JDK 21: Failure due to invalid library name
$ asprof jps load BAD_LIBRARY
Connected to remote JVM
JVM response code = 0
BAD_LIBRARY was not loaded.
libBAD_LIBRARY.so: cannot open shared object file: No such file or directory

// JDK 21: Failure due to -XX:-EnableDynamicAgentLoading
$ asprof jps load BAD_LIBRARY
Connected to remote JVM
JVM response code = 0
Dynamic agent loading is not enabled. Use -XX:+EnableDynamicAgentLoading to launch target VM.

// JDK 21: Success
$ asprof jps load GOOD_LIBRARY
Connected to remote JVM
JVM response code = 0
return code: 0

Note that Dynamic Attach now reports the code '0', no matter if the library load was successful or not.
To work around this, jattach probably needs to parse a text message, which is unfortunate and not very reliable.

[apangin]

I pushed a workaround. Please check if it works for you.

[vans239]

Thanks. Asprof now fails as expected.

$ ~/build2/build/bin/asprof  -d 5 jps
../lib/libasyncProfiler.so was not loaded.
/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ../lib/libasyncProfiler.so)

[apangin]

Submitted a JBS issue: JDK-8325530.