Missing method name from flamegraph

[tosheer]

I am trying to build flamegraph, flamegraph is getting created successfully, but it is having a lot of methods without name / Generic symbol.

Out put which i get is below.

[aemauthor@local-aem62-test bin]$ ./perf-java-flames 17715
Recording events for 15 seconds (adapt by setting PERF_RECORD_SECONDS)
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.696 MB /tmp/perf-17715.data (618 samples) ]
Failed to open 3E, continuing without symbols
Flame graph SVG written to PERF_FLAME_OUTPUT='/home/aemauthor/perf-map-agent/bin/flamegraph-17715.svg'.

Perf data, method map and Flamegraph are attached.

Archive.zip

[zouyx]

@tosheer i have the same problem,

The flame graph which i created is too much unknown item in it，i have no idea what going on。

but it won't like this before，i don't know what my workmate did on production cause this problem。

[nitsanw]

@tosheer Looking at the data you sent it seems like the map file is generate correctly, but perf is unable to correctly map some of the address in your data file anyway. I also notice in your flame graph that a large component of the graph is in compilation. This may explain the issue to some extent if the map is generated after the profiled period, in which some of the code is re-compiled, the addresses will not match.
Can you please give a bit more background on your setup?

• OS
• JDK vendor/version
• JVM options
  Can you also find the git tag for both perf-map-agent and FlameGraph

[ceeaspb]

I've had some problems of my own so taking a look at this also.
@tosheer 's archive zip data shows:

kernel: 3.10.0-123.9.3.el7.x86_64
java: jdk1.8.0_111

the flamegraph flamegraph-17715.svg shows lots of compilation, but there are no such frames in the perf data file you provided. so I guess the 2 don't come from the same recording?

The perf map is being used by perf script and producing what look like reasonable stacks (ie. the frames are not in some random order).

so It looks like perf had some kind of an issue as it has recorded a lot of the frames as having dso "3E". it tries to look up the addresses for this "3E" dso and can't resolve them.
If perf had recorded those frames as coming from the perf map file then it may have been ok.

I took one of these "3E" frame addresses and it does map to an address in the perf map file.
7facfbc09f5a [unknown] (3E)

# grep 7facfbc09 /tmp/perf-17715.map 
7facfbc09740 f60 Lorg/apache/felix/http/sslfilter/internal/SslFilter;::doFilter

and sure enough for some stacks it is recorded/reported correctly:

7facfbc09f5a Lorg/apache/felix/http/sslfilter/internal/SslFilter;::doFilter+0x81a (/tmp/perf-17715.map)

[jrudolph]

@tosheer have you been able to make it work? Would be nice to have an update to figure out what was wrong so we can help people in the future having similar problems. Thanks, @ceeaspb @nitsanw for the analysis.

[jrudolph]

Closing for now.

[wobenhetu]

@tosheer i have the same problem,

The flame graph which i created is too much unknown item in it，i have no idea what going on。

but it won't like this before，i don't know what my workmate did on production cause this problem。

As of Linux 4.6, capturing kernel call stacks using perf_events from a non- root process requires setting two runtime variables. You can set them using sysctl or as follows:

echo 1 > /proc/sys/kernel/perf_event_paranoid

echo 0 > /proc/sys/kernel/kptr_restrict

I have encountered such a problem；That's how it works;