Unexpected EOF reading response | CPU Profiling

[n-vicky]

Hi,

Getting this Unexpected EOF reading response error, while triggering profiling for a linux machine.

Below versions for quick reference,

Async Profiler - async-profiler_2_9

jdk -
openjdk version "1.8.0_362"
OpenJDK Runtime Environment (Zulu 8.68.0.21-CA-linux64) (build 1.8.0_362-b09)
OpenJDK 64-Bit Server VM (Zulu 8.68.0.21-CA-linux64) (build 25.362-b09, mixed mode)

Arguments set -
-XX:+PreserveFramePointer -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints
-XX:MaxJavaStackTraceDepth=1000

P.S - Validated the same in a different machine with the above spec and its working fine.

Kindly help!!
TIA

[apangin]

This message typically means the target JVM has been terminated abnormally.
Check the output of the Java process.

[n-vicky]

@apangin - could observe below error in the logs :

*** Error in `~/jdk/bin/java': free(): invalid pointer: 0x00007f6a993d2c80 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7f7115c82329]
/lib64/libstdc++.so.6(_ZNSt6locale5_Impl16_M_install_facetEPKNS_2idEPKNS_5facetE+0x142)[0x7f6a9913c192]
/lib64/libstdc++.so.6(_ZNSt6locale5_ImplC1Em+0x1e3)[0x7f6a9913c5e3]
/lib64/libstdc++.so.6(+0x71555)[0x7f6a9913d555]
/lib64/libpthread.so.0(+0x620b)[0x7f71161d920b]
/lib64/libstdc++.so.6(+0x715a1)[0x7f6a9913d5a1]
/lib64/libstdc++.so.6(_ZNSt6localeC2Ev+0x13)[0x7f6a9913d5e3]
/lib64/libstdc++.so.6(_ZNSt8ios_base4InitC2Ev+0xbc)[0x7f6a9913a43c]
~/async-profiler_2_9/build/libasyncProfiler.so(+0x38f23)[0x7f6a996bff23]
/lib64/ld-linux-x86-64.so.2(+0xf9c3)[0x7f71163fe9c3]
/lib64/ld-linux-x86-64.so.2(+0x1459e)[0x7f711640359e]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f71163fe7d4]
/lib64/ld-linux-x86-64.so.2(+0x13b8b)[0x7f7116402b8b]
/lib64/libdl.so.2(+0xfab)[0x7f7115fcffab]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f71163fe7d4]
/lib64/libdl.so.2(+0x15ad)[0x7f7115fd05ad]
/lib64/libdl.so.2(dlopen+0x31)[0x7f7115fd0041]
~/jdk/jre/lib/amd64/server/libjvm.so(+0x96bba1)[0x7f711570cba1]
~/jre/lib/amd64/server/libjvm.so(+0x7cfd81)[0x7f7115570d81]
~/jdk/jre/lib/amd64/server/libjvm.so(+0x2ed3af)[0x7f711508e3af]
~/jdk/jre/lib/amd64/server/libjvm.so(+0xae16cb)[0x7f71158826cb]
~/jdk/jre/lib/amd64/server/libjvm.so(+0xae19c1)[0x7f71158829c1]
~/jdk/jre/lib/amd64/server/libjvm.so(+0x96aa72)[0x7f711570ba72]
gs-2023-08-10-03-44-26
/lib64/libpthread.so.0(+0x7ea5)[0x7f71161daea5]
/lib64/libc.so.6(clone+0x6d)[0x7f7115cff9fd]
======= Memory map: ========
3d7c00000-7c1814000 rw-p 00000000 00:00 0
7c1814000-800000000 ---p 00000000 00:00 0 ........

[apangin]

The error happens in libstdc++ when loading async-profiler library.
This must be some incompatibility between the OS version of libstdc++ and the version used to build the profiler.

What OS distribution exactly do you use?

Try to build async-profiler on your system from sources.

[n-vicky]

Below, OS Version, surprisingly in profiler working machines also, the same OS is only available -

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[apangin]

I've just checked async-profiler 2.9 with exactly the same JDK version (Azul Zulu 8.68.0.21) on CentOS 7, and everything worked fine.

Please provide more details on how to reproduce the issue.
Does the error happen every time?
Does async-profiler work if you build it from sources on the same machine?

[n-vicky]

I've just checked async-profiler 2.9 with exactly the same JDK version (Azul Zulu 8.68.0.21) on CentOS 7, and everything worked fine.
--> Yes, it works for me too, in other machines with same configuration, but fails in one machine alone.

Please provide more details on how to reproduce the issue. Does the error happen every time?
--> Error happens every time, immediately once I trigger the profiling.

Does async-profiler work if you build it from sources on the same machine?
--> As the error happens in a Prod sever, I don't have access to work on this. Will check and update you.

[apangin]

Thanks. Please also add the output of grep asyncProfiler /proc/[pid]/maps after the failed attempt.

[n-vicky]

Don't have /proc/[pid]/maps file/folder in our setup. Any other way to debug this issue?

[apangin]

Don't have /proc/[pid]/maps file/folder in our setup

Sorry, I don't understand that.
/proc is a built-in pseudo-filesystem that provides an interface to Linux kernel. It is always available (well, there are ways to hide procfs, but in this case many programs including async-profiler will not work at all).

[n-vicky]

We do have /proc folder, There is pid folder and the files.

[apangin]

So what does the command grep asyncProfiler /proc/PID/maps print?
(here PID is the process ID of the target JVM process)

If the command prints "No such file or directory", it means there is something wrong with procfs on your system.
Async-profiler requires procfs to be accessible.

[n-vicky]

This is the output of the maps file, after failed attempt. Please check, if the below info is sufficient to analyse this issue.

7f1f5d407000-7f1f5d44f000 r-xp 00000000 fd:01 152946 async-profiler_2_9/build/libasyncProfiler.so
7f1f5d44f000-7f1f5d64f000 ---p 00048000 fd:01 152946 async-profiler_2_9/build/libasyncProfiler.so
7f1f5d64f000-7f1f5d651000 rw-p 00048000 fd:01 152946 async-profiler_2_9/build/libasyncProfiler.so

[apangin]

Thanks, the output looks as expected.
I suppose there is an issue with libstdc++ on the host.
To verify this, I've build a version of async-profiler that has libstdc++ linked statically:
async-profiler-3.0-ea-linux-x64.tar.gz
Can you verify if the provided build works on your system?

[apangin]

@n-vicky Do you still experience problems with the provided binary or can I close the issue?

[n-vicky]

@apangin Kindly hold please. I will update you asap !!

[n-vicky]

@apangin

@n-vicky Do you still experience problems with the provided binary or can I close the issue?

Yes, I still face the issue. Below the entire log for your reference.

~/async-profiler_2_9/profiler.sh -e cpu -d 10 -f f1.html 20652
*** Error in `//jdk/bin/java': free(): invalid pointer: 0x00007f65639acd20 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81299)[0x7f694facd299]
/opt//async-profiler_2_9/build/libasyncProfiler.so(_ZNSt6locale5_Impl16_M_install_facetEPKNS_2idEPKNS_5facetE+0x142)[0x7f656371f802]
/opt//async-profiler_2_9/build/libasyncProfiler.so(_ZNSt6locale5_ImplC2Em+0x1e3)[0x7f656370c123]
/opt//async-profiler_2_9/build/libasyncProfiler.so(_ZNSt6locale18_S_initialize_onceEv+0x15)[0x7f656370d095]
/lib64/libpthread.so.0(+0x620b)[0x7f695002420b]
/opt//async-profiler_2_9/build/libasyncProfiler.so(_ZNSt6locale13_S_initializeEv+0x21)[0x7f656370d0e1]
/opt//async-profiler_2_9/build/libasyncProfiler.so(_ZNSt6localeC1Ev+0x13)[0x7f656370d123]
/opt//async-profiler_2_9/build/libasyncProfiler.so(_ZNSt8ios_base4InitC1Ev+0xbc)[0x7f656373a1bc]
/opt//async-profiler_2_9/build/libasyncProfiler.so(+0x50d7b)[0x7f65636cfd7b]
/lib64/ld-linux-x86-64.so.2(+0xf9c3)[0x7f69502499c3]
/lib64/ld-linux-x86-64.so.2(+0x1459e)[0x7f695024e59e]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f69502497d4]
/lib64/ld-linux-x86-64.so.2(+0x13b8b)[0x7f695024db8b]
/lib64/libdl.so.2(+0xfab)[0x7f694fe1afab]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f69502497d4]
/lib64/libdl.so.2(+0x15ad)[0x7f694fe1b5ad]
/lib64/libdl.so.2(dlopen+0x31)[0x7f694fe1b041]
//jdk/jre/lib/amd64/server/libjvm.so(+0x96bba1)[0x7f694f557ba1]
//jdk/jre/lib/amd64/server/libjvm.so(+0x7cfd81)[0x7f694f3bbd81]
//jdk/jre/lib/amd64/server/libjvm.so(+0x2ed3af)[0x7f694eed93af]
//jdk/jre/lib/amd64/server/libjvm.so(+0xae16cb)[0x7f694f6cd6cb]
//jdk/jre/lib/amd64/server/libjvm.so(+0xae19c1)[0x7f694f6cd9c1]
//jdk/jre/lib/amd64/server/libjvm.so(+0x96aa72)[0x7f694f556a72]
/lib64/libpthread.so.0(+0x7ea5)[0x7f6950025ea5]
/lib64/libc.so.6(clone+0x6d)[0x7f694fb4a96d]
======= Memory map: ========
6cbc00000-7c1ae0000 rw-p 00000000 00:00 0
7c1ae0000-800000000 ---p 00000000 00:00 0
561d4a316000-561d4a317000 r-xp 00000000 fc:00 6425588 //jdk/bin/java
561d4a317000-561d4a318000 r--p 00000000 fc:00 6425588 //jdk/bin/java
561d4a318000-561d4a319000 rw-p 00001000 fc:00 6425588 //jdk/bin/java
561d4a319000-561d4a35e000 rw-p 00000000 00:00 0 [heap]
7f6558260000-7f6558275000 r-xp 00000000 fd:02 260119 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f6558275000-7f6558474000 ---p 00015000 fd:02 260119 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f6558474000-7f6558475000 r--p 00014000 fd:02 260119 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f6558475000-7f6558476000 rw-p 00015000 fd:02 260119 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f6558476000-7f655847a000 ---p 00000000 00:00 0
7f655847a000-7f6558577000 rw-p 00000000 00:00 0
7f6558577000-7f655857b000 ---p 00000000 00:00 0
7f655857b000-7f6558678000 rw-p 00000000 00:00 0
7f6558678000-7f655867c000 ---p 00000000 00:00 0
7f655867c000-7f6558779000 rw-p 00000000 00:00 0
7f6558779000-7f655877d000 ---p 00000000 00:00 0
7f655877d000-7f655887a000 rw-p 00000000 00:00 0
7f655887a000-7f655887e000 ---p 00000000 00:00 0
7f655887e000-7f655897b000 rw-p 00000000 00:00 0
7f655897b000-7f655897f000 ---p 00000000 00:00 0
7f655897f000-7f6558a7c000 rw-p 00000000 00:00 0
7f6558a7c000-7f6558a80000 ---p 00000000 00:00 0
7f6558a80000-7f6558bfd000 rw-p 00000000 00:00 0
7f6558bfd000-7f6558d7d000 ---p 00000000 00:00 0
7f6558d7d000-7f6558d81000 ---p 00000000 00:00 0
7f6558d81000-7f6558e7e000 rw-p 00000000 00:00 0
7f6558e7e000-7f6558e82000 ---p 00000000 00:00 0
7f6558e82000-7f6558f7f000 rw-p 00000000 00:00 0
7f6558f7f000-7f6558f83000 ---p 00000000 00:00 0
7f6558f83000-7f6559080000 rw-p 00000000 00:00 0
7f6559080000-7f6559084000 ---p 00000000 00:00 0
7f6559084000-7f6559181000 rw-p 00000000 00:00 0
7f6559181000-7f6559185000 ---p 00000000 00:00 0
7f6559185000-7f6559282000 rw-p 00000000 00:00 0
7f6559282000-7f6559286000 ---p 00000000 00:00 0
7f6559286000-7f6559383000 rw-p 00000000 00:00 0
7f6559383000-7f6559387000 ---p 00000000 00:00 0
7f6559387000-7f6559484000 rw-p 00000000 00:00 0
7f6559484000-7f6559488000 ---p 00000000 00:00 0
7f6559488000-7f6559585000 rw-p 00000000 00:00 0
7f6559585000-7f6559589000 ---p 00000000 00:00 0
7f6559589000-7f6559686000 rw-p 00000000 00:00 0
7f6559686000-7f655968a000 ---p 00000000 00:00 0
7f655968a000-7f6559787000 rw-p 00000000 00:00 0
7f6559787000-7f655978b000 ---p 00000000 00:00 0
7f655978b000-7f6559888000 rw-p 00000000 00:00 0
7f6559888000-7f655988c000 ---p 00000000 00:00 0
7f655988c000-7f6559989000 rw-p 00000000 00:00 0
7f6559989000-7f655998d000 ---p 00000000 00:00 0
7f655998d000-7f6559a8a000 rw-p 00000000 00:00 0
7f6559b8b000-7f6559b8f000 ---p 00000000 00:00 0
7f6559b8f000-7f6559c8c000 rw-p 00000000 00:00 0
7f6559c8c000-7f6559c90000 ---p 00000000 00:00 0
7f6559c90000-7f6559d8d000 rw-p 00000000 00:00 0
7f6559d8d000-7f6559d91000 ---p 00000000 00:00 0
7f6559d91000-7f6559e8e000 rw-p 00000000 00:00 0
7f6559e8e000-7f6559e92000 ---p 00000000 00:00 0
7f6559e92000-7f6559f8f000 rw-p 00000000 00:00 0
7f6559f8f000-7f6559f93000 ---p 00000000 00:00 0
7f6559f93000-7f655a090000 rw-p 00000000 00:00 0
7f655a090000-7f655a094000 ---p 00000000 00:00 0
7f655a094000-7f655a191000 rw-p 00000000 00:00 0
7f655a191000-7f655a195000 ---p 00000000 00:00 0
7f655a195000-7f655a292000 rw-p 00000000 00:00 0
7f655a292000-7f655a296000 ---p 00000000 00:00 0
7f655a296000-7f655a393000 rw-p 00000000 00:00 0
7f655a393000-7f655a397000 ---p 00000000 00:00 0
7f655a397000-7f655a494000 rw-p 00000000 00:00 0
7f655a494000-7f655a498000 ---p 00000000 00:00 0
7f655a498000-7f655a595000 rw-p 00000000 00:00 0
7f655a595000-7f655a599000 ---p 00000000 00:00 0
7f655a599000-7f655a696000 rw-p 00000000 00:00 0
7f655a696000-7f655a69a000 ---p 00000000 00:00 0
7f655a69a000-7f655a797000 rw-p 00000000 00:00 0
7f655a797000-7f655a79b000 ---p 00000000 00:00 0
7f655a79b000-7f655b899000 rw-p 00000000 00:00 0
7f655b899000-7f655b89d000 ---p 00000000 00:00 0
7f655b89d000-7f655c99b000 rw-p 00000000 00:00 0
7f655c99b000-7f655c99f000 ---p 00000000 00:00 0
7f655c99f000-7f655da9d000 rw-p 00000000 00:00 0
7f655da9d000-7f655daa1000 ---p 00000000 00:00 0
7f655daa1000-7f655db9e000 rw-p 00000000 00:00 0
7f655db9e000-7f655dba2000 ---p 00000000 00:00 0
7f655dba2000-7f655dc9f000 rw-p 00000000 00:00 0
7f655dc9f000-7f655dca3000 ---p 00000000 00:00 0
7f655dca3000-7f655eda1000 rw-p 00000000 00:00 0
7f655eda1000-7f655eda5000 ---p 00000000 00:00 0
7f655eda5000-7f655eea2000 rw-p 00000000 00:00 0
7f655eea2000-7f655eea6000 ---p 00000000 00:00 0
7f655eea6000-7f655efa3000 rw-p 00000000 00:00 0
7f655efa3000-7f655efa7000 ---p 00000000 00:00 0
7f655efa7000-7f655f0a4000 rw-p 00000000 00:00 0
7f655f0a4000-7f655f0a8000 ---p 00000000 00:00 0
7f655f0a8000-7f655f3a5000 rw-p 00000000 00:00 0
7f655f3a5000-7f65633a9000 rw-p 00000000 00:00 0
7f65633a9000-7f65633ad000 ---p 00000000 00:00 0
7f65633ad000-7f65634aa000 rw-p 00000000 00:00 0
7f65634aa000-7f65634ae000 ---p 00000000 00:00 0
7f65634ae000-7f65635ab000 rw-p 00000000 00:00 0
7f656367f000-7f6563788000 r-xp 00000000 fd:02 149207 /opt//async-profiler_2_9/build/libasyncProfiler.so
7f6563788000-7f6563988000 ---p 00109000 fd:02 149207 /opt//async-profiler_2_9/build/libasyncProfiler.so
7f6563988000-7f656398f000 r--p 00109000 fd:02 149207 /opt//async-profiler_2_9/build/libasyncProfiler.so
7f656398f000-7f6563992000 rw-p 00110000 fd:02 149207 /opt/****/async-profiler_2_9/build/libasyncProfiler.so
7f6563992000-7f65639af000 rw-p 00000000 00:00 0
7f65639af000-7f65639b3000 ---p 00000000 00:00 0
7f65639b3000-7f6563ab0000 rw-p 00000000 00:00 0
7f6563ab0000-7f6563ab4000 ---p 00000000 00:00 0
7f6563ab4000-7f6563bb1000 rw-p 00000000 00:00 0
7f6563bb1000-7f6563bb5000 ---p 00000000 00:00 0
7f6563bb5000-7f6563cb2000 rw-p 00000000 00:00 0
7f6563cb2000-7f6563cb6000 ---p 00000000 00:00 0
7f6563cb6000-7f6563db3000 rw-p 00000000 00:00 0
7f6563db3000-7f6563db7000 ---p 00000000 00:00 0
7f6563db7000-7f6563eb4000 rw-p 00000000 00:00 0
7f6563eb4000-7f6563eb8000 ---p 00000000 00:00 0
7f6563eb8000-7f6563fb5000 rw-p 00000000 00:00 0
7f6563fb5000-7f6563fb9000 ---p 00000000 00:00 0
7f6563fb9000-7f65640b6000 rw-p 00000000 00:00 0
7f65640b6000-7f65640ba000 ---p 00000000 00:00 0
7f65640ba000-7f65641b7000 rw-p 00000000 00:00 0
7f65641b7000-7f65641bb000 ---p 00000000 00:00 0
7f65641bb000-7f65642b8000 rw-p 00000000 00:00 0
7f65642b8000-7f65642bc000 ---p 00000000 00:00 0
7f65642bc000-7f65643b9000 rw-p 00000000 00:00 0
7f65643b9000-7f65643bd000 ---p 00000000 00:00 0
7f65643bd000-7f65644ba000 rw-p 00000000 00:00 0
7f65644ba000-7f65644be000 ---p 00000000 00:00 0
7f65644be000-7f65645bb000 rw-p 00000000 00:00 0
7f65645bb000-7f65645bf000 ---p 00000000 00:00 0
7f65645bf000-7f65646bc000 rw-p 00000000 00:00 0
7f65646bc000-7f65646c0000 ---p 00000000 00:00 0
7f65646c0000-7f65647bd000 rw-p 00000000 00:00 0
7f65647bd000-7f65647c1000 ---p 00000000 00:00 0
7f65647c1000-7f65648be000 rw-p 00000000 00:00 0
7f65648be000-7f65648c2000 ---p 00000000 00:00 0
7f65648c2000-7f65649bf000 rw-p 00000000 00:00 0
7f65649bf000-7f65649c3000 ---p 00000000 00:00 0
7f65649c3000-7f6564ac0000 rw-p 00000000 00:00 0
7f6564ac0000-7f6564ac4000 ---p 00000000 00:00 0
7f6564ac4000-7f6564bc1000 rw-p 00000000 00:00 0
7f6564bc1000-7f6564bc5000 ---p 00000000 00:00 0
7f6564bc5000-7f6564cc2000 rw-p 00000000 00:00 0
7f6564cc2000-7f6564cc6000 ---p 00000000 00:00 0
7f6564cc6000-7f6564dc3000 rw-p 00000000 00:00 0
7f6564dc3000-7f6564dc7000 ---p 00000000 00:00 0
7f6564dc7000-7f6564ec4000 rw-p 00000000 00:00 0
7f6564ec4000-7f6564ec8000 ---p 00000000 00:00 0
7f6564ec8000-7f6564fc5000 rw-p 00000000 00:00 0
7f6564fc5000-7f6564fc9000 ---p 00000000 00:00 0
7f6564fc9000-7f65650c6000 rw-p 00000000 00:00 0
7f65650c6000-7f65650ca000 ---p 00000000 00:00 0
7f65650ca000-7f65651c7000 rw-p 00000000 00:00 0
7f65651c7000-7f65651cb000 ---p 00000000 00:00 0
7f65651cb000-7f65652c8000 rw-p 00000000 00:00 0
7f65652c8000-7f65652cc000 ---p 00000000 00:00 0
7f65652cc000-7f65653c9000 rw-p 00000000 00:00 0
7f65653c9000-7f65653cd000 ---p 00000000 00:00 0
7f65653cd000-7f65654ca000 rw-p 00000000 00:00 0
7f65654ca000-7f65654ce000 ---p 00000000 00:00 0
7f65654ce000-7f65655cb000 rw-p 00000000 00:00 0
7f65655cb000-7f65655cf000 ---p 00000000 00:00 0
7f65655cf000-7f65656cc000 rw-p 00000000 00:00 0
7f65656cc000-7f65656d0000 ---p 00000000 00:00 0
7f65656d0000-7f65657cd000 rw-p 00000000 00:00 0
7f65657cd000-7f65657d1000 ---p 00000000 00:00 0
7f65657d1000-7f65658ce000 rw-p 00000000 00:00 0
7f65658ce000-7f65658d2000 ---p 00000000 00:00 0
7f65658d2000-7f65659cf000 rw-p 00000000 00:00 0
7f65659cf000-7f65659d3000 ---p 00000000 00:00 0
7f65659d3000-7f6565ad0000 rw-p 00000000 00:00 0
7f6565ad0000-7f6565ad4000 ---p 00000000 00:00 0
7f6565ad4000-7f6565bd1000 rw-p 00000000 00:00 0
7f6565bd1000-7f6565bd5000 ---p 00000000 00:00 0
7f6565bd5000-7f6565cd2000 rw-p 00000000 00:00 0
7f6565cd2000-7f6565cd6000 ---p 00000000 00:00 0
7f6565cd6000-7f6565dd3000 rw-p 00000000 00:00 0
7f6565dd3000-7f6565dd7000 ---p 00000000 00:00 0
7f6565dd7000-7f6566ed5000 rw-p 00000000 00:00 0
7f6566ed5000-7f6566ed9000 ---p 00000000 00:00 0
7f6566ed9000-7f6567fd7000 rw-p 00000000 00:00 0
7f6567fd7000-7f6567fdb000 ---p 00000000 00:00 0
7f6567fdb000-7f65680d8000 rw-p 00000000 00:00 0
7f65680d8000-7f65680dc000 ---p 00000000 00:00 0
7f65680dc000-7f65681d9000 rw-p 00000000 00:00 0
7f65681d9000-7f65681dd000 ---p 00000000 00:00 0
7f65681dd000-7f65682da000 rw-p 00000000 00:00 0
7f65682da000-7f65682de000 ---p 00000000 00:00 0
7f65682de000-7f65683db000 rw-p 00000000 00:00 0
7f65683db000-7f65683df000 ---p 00000000 00:00 0
7f65683df000-7f65684dc000 rw-p 00000000 00:00 0
7f65684dc000-7f65684e0000 ---p 00000000 00:00 0
7f65684e0000-7f65685dd000 rw-p 00000000 00:00 0
7f65685dd000-7f65685e1000 ---p 00000000 00:00 0
7f65685e1000-7f65686de000 rw-p 00000000 00:00 0
7f65686de000-7f65686e2000 ---p 00000000 00:00 0
7f65686e2000-7f65687df000 rw-p 00000000 00:00 0
7f65687df000-7f65687e3000 ---p 00000000 00:00 0
7f65687e3000-7f65688e0000 rw-p 00000000 00:00 0
7f65688e0000-7f65688e4000 ---p 00000000 00:00 0
7f65688e4000-7f65689e1000 rw-p 00000000 00:00 0
7f65689e1000-7f65689e5000 ---p 00000000 00:00 0
7f65689e5000-7f6568ae2000 rw-p 00000000 00:00 0
7f6568ae2000-7f6568ae6000 ---p 00000000 00:00 0
7f6568ae6000-7f6568be3000 rw-p 00000000 00:00 0
7f6568be3000-7f6568be7000 ---p 00000000 00:00 0
7f6568be7000-7f6568ce4000 rw-p 00000000 00:00 0
7f6568ce4000-7f6568ce8000 ---p 00000000 00:00 0
7f6568ce8000-7f6568de5000 rw-p 00000000 00:00 0
7f6568de5000-7f6568de9000 ---p 00000000 00:00 0
7f6568de9000-7f6568ee6000 rw-p 00000000 00:00 0
7f6568ee6000-7f6568eea000 ---p 00000000 00:00 0
7f6568eea000-7f6568fe7000 rw-p 00000000 00:00 0
7f6568fe7000-7f6568feb000 ---p 00000000 00:00 0
7f6568feb000-7f65690e8000 rw-p 00000000 00:00 0
7f65690e8000-7f65690ec000 ---p 00000000 00:00 0
7f65690ec000-7f65691e9000 rw-p 00000000 00:00 0
7f65691e9000-7f65691ed000 ---p 00000000 00:00 0
7f65691ed000-7f65692ea000 rw-p 00000000 00:00 0
7f65692ea000-7f65692ee000 ---p 00000000 00:00 0
7f65692ee000-7f65693eb000 rw-p 00000000 00:00 0
7f65693eb000-7f65693ef000 ---p 00000000 00:00 0
7f65693ef000-7f65694ec000 rw-p 00000000 00:00 0
7f65694ec000-7f65694f0000 ---p 00000000 00:00 0
7f65694f0000-7f65695ed000 rw-p 00000000 00:00 0
7f65695ed000-7f65695f1000 ---p 00000000 00:00 0
7f65695f1000-7f65696ee000 rw-p 00000000 00:00 0
7f65696ee000-7f65696f2000 ---p 00000000 00:00 0
7f65696f2000-7f65697ef000 rw-p 00000000 00:00 0
7f65697ef000-7f65697f3000 ---p 00000000 00:00 0
7f65697f3000-7f65698f0000 rw-p 00000000 00:00 0
7f65698f0000-7f65698f4000 ---p 00000000 00:00 0
7f65698f4000-7f65699f1000 rw-p 00000000 00:00 0
7f65699f1000-7f65699f5000 ---p 00000000 00:00 0
7f65699f5000-7f6569af2000 rw-p 00000000 00:00 0
7f6569af2000-7f6569af6000 ---p 00000000 00:00 0
7f6569af6000-7f6569bf3000 rw-p 00000000 00:00 0
7f6569bf3000-7f6569bf7000 ---p 00000000 00:00 0
7f6569bf7000-7f6569cf4000 rw-p 00000000 00:00 0
7f6569cf4000-7f6569cf8000 ---p 00000000 00:00 0
7f6569cf8000-7f6569df5000 rw-p 00000000 00:00 0
7f6569df5000-7f6569df9000 ---p 00000000 00:00 0
7f6569df9000-7f6569ef6000 rw-p 00000000 00:00 0
7f6569ef6000-7f6569efa000 ---p 00000000 00:00 0
7f6569efa000-7f6569ff7000 rw-p 00000000 00:00 0
7f6569ff7000-7f6569ffb000 ---p 00000000 00:00 0
7f6569ffb000-7f656a0f8000 rw-p 00000000 00:00 0
7f656a0f8000-7f656a0fc000 ---p 00000000 00:00 0
7f656a0fc000-7f656a1f9000 rw-p 00000000 00:00 0
7f656a1f9000-7f656a1fd000 ---p 00000000 00:00 0
7f656a1fd000-7f656a2fa000 rw-p 00000000 00:00 0
7f656a2fa000-7f656a2fe000 ---p 00000000 00:00 0
7f656a2fe000-7f656a3fb000 rw-p 00000000 00:00 0
7f656a3fb000-7f656a3ff000 ---p 00000000 00:00 0
7f656a3ff000-7f656a4fc000 rw-p 00000000 00:00 0
7f656a4fe000-7f656a6fe000 rw-p 00000000 00:00 0
7f656a6fe000-7f656a702000 ---p 00000000 00:00 0
7f656a702000-7f656a7ff000 rw-p 00000000 00:00 0
7f656a7ff000-7f656a803000 ---p 00000000 00:00 0
7f656a803000-7f656a900000 rw-p 00000000 00:00 0
7f656a900000-7f656a904000 ---p 00000000 00:00 0
7f656a904000-7f656aa01000 rw-p 00000000 00:00 0
7f656aa01000-7f656aa05000 ---p 00000000 00:00 0
7f656aa05000-7f656ab02000 rw-p 00000000 00:00 0
7f656ab02000-7f656ab06000 ---p 00000000 00:00 0
7f656ab06000-7f656ac03000 rw-p 00000000 00:00 0
7f656ac03000-7f656ac07000 ---p 00000000 00:00 0
7f656ac07000-7f656ad04000 rw-p 00000000 00:00 0
7f656ad04000-7f656ad08000 ---p 00000000 00:00 0
7f656ad08000-7f656ae05000 rw-p 00000000 00:00 0
7f656ae05000-7f656ae09000 ---p 00000000 00:00 0
7f656ae09000-7f656af06000 rw-p 00000000 00:00 0 Unexpected EOF reading response

[apangin]

Does the error happen only with a specific application or with any trivial program?
Does java -agentpath:/path/to/libasyncProfiler.so=start -version work?

[n-vicky]

@apangin

We are running this for a specific application (Tomcat Server).

[apangin]

I assume some of the loaded libraries is linked with a different version of libstdc++ that conflicts with the system version of libstdc++ on CentOS 7.

JDK-8254702 describes a similar issue (also on CentOS 7).

Does the error persist if you add -agentpath:/path/to/libasyncProfiler.so java command line option?
This will force async-profiler library load before other libraries. You can then use asprof/profiler.sh as usual.

[apangin]

I'm closing the issue in the lack of additional information.

Ultimately, #872 should solve all compatibility issues related to libstdc++.

[n-vicky]

@apangin Apologise for not providing an update on this !! Multiple teams' approval to check this fix in pre-prod, eating time.

I will apprise you asap !!

Thank you for your prompt replies on my queries, appreciate it.