Improve reporting of inlined frames

[nitsanw]

I think PMA is in a unique position to break new grounds in Java profiling by reporting the Java stack while highlighting the separation between real frames and virtual frames(inlined methods). To do this we will need:

1. Report all frames when reporting inlined data: This can easily lead to VERY long strings. I'm not sure what the maximum 'method' name length supported by perf is.
2. Support from flame graphs in colouring and displaying the frames.

[jrudolph]

Yes, that's something I'd like to do as well. I wonder if it makes sense at all to rely on the perf user-space tools to do the symbol lookup or if it wouldn't make more sense to implement the expansion in an extra tool.

[nitsanw]

It adds up to much the same process perf is doing anyhow, matching a range of addresses to a string. There's an optimization to be had by logging the jmethodIds rather than the long string they represent. That might be helpful for FlameGraph tooling, but will not help perf-top etc.

[jrudolph]

Yes, but this is slightly different as we match one address to a whole stack of frames which is nothing that perf tools currently support.

[nitsanw]

True, I thought we could just make it a really long string and unwind it in FlameGraphs

[jrudolph]

The more I think about it the more I like your simple suggestion.

[jrudolph]

Here's a first approximation:

https://github.com/jrudolph/perf-map-agent/tree/w/25-output-all-inlined-frames

[jrudolph]

I'm using this branch now for a while and it seems to work basically as intended.

I recently noticed an odd thing, though: it seems that the reported information for addresses which contain inlined code are not as granular as they are reported with PrintAssembly. I haven't yet researched deeply why that would be the case.

FTR here's OpenJDK code which produces the inline info: http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/87ee5ee27509/src/share/vm/prims/jvmtiExport.cpp#l1716

and here the code which generates code annotations for PrintAssembly: http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/87ee5ee27509/src/share/vm/code/nmethod.cpp#l2807

The effect of the problem is that the most interesting addresses in a JIT-compiled method that contains all the inlined stuff from other methods are reported as if they belonged to the JIT-compiled method (and not to the inlined ones). On the hand, it seems that code that is generated as the slow fallback will be properly annotated.

[nitsanw]

Are you enabling non safepoint debug info?

[jrudolph]

No, I wasn't! Adding -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints I'll get inlining info for the rest of the addresses as well.

Thanks, @nitsanw.

[nitsanw]

a classic :-) catches me out every once in a while too

[nitsanw]

Are you planning on merging the branch?

[jrudolph]

It's finished from my side, so if anyone wants to have a look, see #35 which I will soon merge.