Not able to resolve symbols

[luan-cestari]

Hi

I'm trying to use your project on a fedora 24 using openjdk 1.8.0.101 . The problem can be seen using perf-java-top and flame graphs as well . The create-java-perf-map seems to work well but it did saw how could get its output with perf to plot with frame. Jstack works as well (but as you mention it would be a terrible poor performance and quality).

I thought it would be something with preserving the framepointer but the jdk version is compatible and I'm using the following JVM options:

-XX:+PreserveFramePointer -XX:+StartAttachListener -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:-OmitStackTraceInFastThrow -XX:+ShowHiddenFrames

Do you have any suggestions about how to solve this problem?

Thanks in advance
Luan

[jrudolph]

@luan-cestari I'm not sure I understand. Could you rephrase your question or add some concrete information which output is incorrect? Does the agent generate the /tmp/perf-.map file? What does it contain?

[luan-cestari]

Hi @jrudolph

Thank you very much for the prompt reply.

The problem that I got is when using some tools such as perf-java-top there are many libjvm.so Objects that point to some hexadecimal numbers (memory address probably) which instead I would expect to see the java methods called. Below a snippet of the perf-java-top output for example:

bin/perf-java-top $PID

Samples: 101  of event 'cycles:ppp', Event count (approx.): 33443277
Overhead  Shared Object       Symbol
  10.15%  perf-25647.map      [.] Interpreter
   6.17%  libjvm.so           [.] 0x0000000000a3c103
   6.09%  libc-2.23.so        [.] __clock_gettime
   5.42%  libjvm.so           [.] 0x00000000005f03c1
   5.03%  libjvm.so           [.] 0x000000000095697a
   4.14%  [kernel]            [k] get_futex_key
   3.79%  libjvm.so           [.] 0x0000000000a05010
   3.78%  libjvm.so           [.] 0x0000000000a6fcbf
   3.34%  libjvm.so           [.] 0x000000000032e1e1
   2.77%  [kernel]            [k] futex_wake
   2.46%  libjvm.so           [.] 0x0000000000a01792
   2.25%  perf-25647.map      [.] Ljava/lang/Integer;::getChars
   2.08%  libjvm.so           [.] 0x000000000056b988
   2.07%  libjvm.so           [.] 0x00000000009bf5bb
   1.87%  libjvm.so           [.] 0x0000000000a4fd68
   1.87%  libpthread-2.23.so  [.] __pthread_mutex_unlock_usercnt
   1.84%  libjvm.so           [.] 0x000000000056c2fd
   1.76%  [kernel]            [k] __schedule
   1.57%  libjvm.so           [.] 0x00000000008a3392
   1.47%  libjvm.so           [.] 0x00000000007bfe0e
   1.42%  libjvm.so           [.] 0x000000000043f490
   1.34%  libjvm.so           [.] 0x0000000000a58204
   1.29%  libjvm.so           [.] 0x00000000007be620
   1.27%  libjvm.so           [.] 0x00000000003d17c6
   ...

In Flame Graphs I see the same problem, almost all the java stack are just "libjvm.so" , instead the name of the Java class and the method invoked.

I think I'm using the right JVM options and a compatible JVM version , as I mentioned in the previous comment.

I tested to use perf-java-top with my intellij IDE (which basically is a java process as well) and it seems to work fine (the Symbols are all resolved to java classes and methods) . So I think it might be something in my project or in jvm that I'm missing to configure.

Did you get something similar to this issue before?

Thank you again for the help.

Kind regards,
Luan

[jrudolph]

libjvm.so is the JVM runtime and not your code. You need to install debug symbols for the JVM to get symbols for these. Often, it's either the JIT compiler or GC that will then show up. When libjvm entries shows up it's usually because either your program is still fresh and still needs lots of JIT compilation or it just does little CPU intensive stuff.

Java code will show up under "perf-25647.map", in your example the JVM Interpreter is shown and Ljava/lang/Integer;::getChars. High interpreter numbers are often an indication that your code is not yet hot. So, if that's the case, try some warmup period before measuring results (as with any JVM benchmark).

[nitsanw]

For reference, I have no issue with seeing libjvm.so symbols on Oracle JDK 1.8 update 92, e.g.:

Samples: 571  of event 'cycles:pp', Event count (approx.): 150293896
Overhead  Shared Object               Symbol
   3.59%  perf-7913.map               [.] Lorg/eclipse/emf/ecore/util/EcoreUtil$
   2.96%  perf-7913.map               [.] Lorg/eclipse/emf/ecore/util/EcoreUtil$
   2.47%  perf-7913.map               [.] Lorg/eclipse/e4/ui/internal/workbench/
   2.44%  libjvm.so                   [.] nmethod::cleanup_inline_caches
   1.91%  perf-7913.map               [.] Lorg/eclipse/swt/widgets/Display;::sle
   1.76%  libjvm.so                   [.] IndexSetIterator::advance_and_next
   1.65%  perf-7913.map               [.] Ljava/util/HashMap;::resize

This is not a special build, but is a JDK, not a JRE download. It might be a build specific issue.

[luan-cestari]

@jrudolph Sorry, I forgot to mention that I'm already using the package that comes with debug symbols (or at least it should be). I agree with you that the program is perhaps didn't have enough time to warm up. I will do some more tests here.

@nitsanw yeah, I was thinking something like that. Do you know someone from openjdk community that could confirm and address and issue there? I don't have permission to open issues in their issue tracker

[leogomes]

$ sudo apt install openjdk-8-dbg worked like a charm for me :)

[ceeaspb]

@luan-cestari @jrudolph rightly said "You need to install debug symbols for the JVM". So to hopefully progress it to closure here is the detail on JVM debug symbols on Fedora 24.

So to get this working as below on fedora 24 (I didn't run PMA here as it's not relevant to the qn):

#perf script|less

...
java  5162 [000]  2175.316177:     250000 cpu-clock: 
                  8c9ab6 os::javaTimeMillis+0xffff014797a64026 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-3.b16.fc24.x86_64/jre/lib/amd64/server/libjvm.so)
            7f5c1d10ec72 [unknown] (/tmp/perf-5152.map)
            7f5c1d0004e7 [unknown] (/tmp/perf-5152.map)
                  670663 JavaCalls::call_helper+0xffff014797a64f53 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-3.b16.fc24.x86_64/jre/lib/amd64/server/libjvm.so)
                  66dbd7 JavaCalls::call_virtual+0xffff014797a64297 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-3.b16.fc24.x86_64/jre/lib/amd64/server/libjvm.so)
                  66e1b7 JavaCalls::call_virtual+0xffff014797a64057 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-3.b16.fc24.x86_64/jre/lib/amd64/server/libjvm.so)
                  6b1814 thread_entry+0xffff014797a64084 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-3.b16.fc24.x86_64/jre/lib/amd64/server/libjvm.so)
                  a46b6b JavaThread::thread_main_inner+0xffff014797a641eb (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-3.b16.fc24.x86_64/jre/lib/amd64/server/libjvm.so)
                  a4704f JavaThread::run+0xffff014797a6448f (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-3.b16.fc24.x86_64/jre/lib/amd64/server/libjvm.so)
                  8c8ed2 java_start+0xffff014797a640f2 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-3.b16.fc24.x86_64/jre/lib/amd64/server/libjvm.so)
                    75ba start_thread+0xffff01479477e0ca (/usr/lib64/libpthread-2.23.so)
...

You need to install debuginfo java.

dnf debuginfo-install java

which installs the debug symbols into /usr/lib/debug/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-3.b16.fc24.x86_64/jre/lib/amd64/libjava.so.debug or similar location. See https://fedoraproject.org/wiki/Packaging:Debuginfo

After that "perf script" works for me as above.
"perf top -g" showed mangled/incomplete stacks (even though "perf script" stacks were OK), so probably worth avoiding "perf top -g" on this perf/os combo.

# perf --version
perf version 4.8.10.200.fc24.x86_64.gc23c
# uname -a
Linux localhost.localdomain 4.5.5-300.fc24.x86_64 #1 SMP Thu May 19 13:05:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release 
Fedora release 24 (Twenty Four)

Running the following does not install debug symbols on fedora:

#dnf install java-debug
...

#objdump -t /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-3.b16.fc24.x86_64-debug/jre/lib/amd64/server/libjvm.so

/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-3.b16.fc24.x86_64-debug/jre/lib/amd64/server/libjvm.so:     file format elf64-x86-64

SYMBOL TABLE:
no symbols

[jrudolph]

Thanks for all the information. I added a small note to the README to look into this issue for pointers about how to install debug symbols for the JVM.

[shizukanaskytree]

I try to use uprobe to trace JVM functions but I get ERROR: missing symbol. Does anyone know how to solve this problem?

root@wxf:/home/wxf/perf-tool/perf-tools/bin# ./uprobe -s p:/home/wxf/9dev/build/linux-x86_64-normal-server-slowdebug/jdk/bin/java:AdaptiveSizePolicy::minor_collection_begin
ERROR: missing symbol "minor_collection_begin" in /home/wxf/9dev/build/linux-x86_64-normal-server-slowdebug/jdk/bin/java

So I try again

root@wxf:/home/wxf/perf-tool/perf-tools/bin# ./uprobe -s p:/home/wxf/9dev/build/linux-x86_64-normal-server-slowdebug/jdk/bin/java:minor_collection_begin
ERROR: missing symbol "minor_collection_begin" in /home/wxf/9dev/build/linux-x86_64-normal-server-slowdebug/jdk/bin/java