Graph View and Flame graph shows Unknown Java Frame

[sattishv]

Hi team,

The profiles collected from cloud profiler java agent with the below listed JVM options shows Unknown java frame in Graph as well as Flame Graph view. The java agent is attached to an microservice which resides inside a docker container.
JVM options used:

-XX:+PreserveFramePointer -agentpath:/opt/profiler_java_agent.so=-cprof_service=UBDS,-cprof_profile_filename=userbridgedomainserv/cronus/scripts/app/logs/profile_,-cprof_interval_sec=5,-cprof_duration_sec=800

Platform: Linux 4.4.0-133-generic #159~14.04.1-Ubuntu SMP
Oracle JVM : 1.8 update 60

[steveniemitz]

+1 on this, additionally, it seems to have issues reconstructing the full thread stack correctly. In my profiles I see lots of frames as the "roots" that definitely aren't (since the roots should really only be Thread.run and similar).

[nolanmar511]

@steveniemitz -- What kind of stack depths do you see in your profiles (# of frames in a call stack)?

For the frames that look like they're "roots" but cannot be, I wonder the application might have a stack depth greater than the default maximum stack depth that the profile will record.

[steveniemitz]

yep! I just stumbled upon the 128 max-frame limit in the profiler. I'm currently rebuilding with a larger limit to see if that fixes it. We definitely have stacks > 128 frames.

[nolanmar511]

I'm less sure about the potential reasons for Unknown Java frames (it could mean stack unwinding failed, it could be that native frames were running)

@jcbeyler for additional insight there.

[steveniemitz]

We're using netty with native epoll, I wonder if the unknown frame is epoll wait and the profiler can't unwind past it. Once I get the frame limit higher I'll start checking that out.

…

On Thu, May 9, 2019 at 12:23 PM Margaret Nolan ***@***.***> wrote: I'm less sure about the potential reasons for Unknown Java frames (it could mean stack unwinding failed, it could be that native frames were running) +jcbeyler@ for additional insight there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOLWZIFZTEMDCKUSB2B5N3PURFXTANCNFSM4GU4HUWA> .

[steveniemitz]

in sort-of-related news...compiling this in docker is really rough. Building protobuf uses 16+ GB of RAM (the max I've allocated to my docker VM) and them OOMs. :(

edit: oof, the build script passes just a bare -j to make, which is unbounded parallelism. I'll try setting a sane value there to see if that helps.

[jcbeyler]

The unknown is that AsyncGetCallTrace cannot walk past a frame and got stuck, I'm slowly starting to add tests for AGCT and then should be able to work on opening that door a bit

[steveniemitz]

alriiiight, bumping kMaxFramesToCapture fixed my issue! As for the "unknown java frame" problem, I still have it showing up a little bit, and am going to try to debug that a little bit now.

My initial theory of it being netty epoll seems disproven, I see epoll_wait, readAddress, and writevAddresses native frame all in the profile.

[jcbeyler]

For the unknown java frame; basically if the stack trace grabber fails, it returns a "this is an unknown Java frame" or a "can't walk the stack trace" error code that the profiler transforms into this. There are known reasons for this; you can solve them sometimes by refactoring your code a bit but that might not always be possible if you are using libraries provoking this

[steveniemitz]

I wonder if its worth investigating using async profiler (or making it an option) as the profiling backend rather than the google one it bundles? It seems like async-profiler has a workaround for this already built in.

[steveniemitz]

alright, so I hacked in some of the logic from async-profiler into the profiling code here, specifically the part that attempts to recover the top frame from the stack manually. My time in "unknown java frame" went from ~60% down to ~10%. There are still some stacks that can't be recovered, but I also didn't port over the full recovery logic from async-profiler. As a POC though this seems really promising.

[aalexand]

On the heavy build issue, one thing to explore is getting rid of the gRPC dependency and send the profiles just over the HTTPS wire.

[steveniemitz]

My build issues were fixed by setting -j4 on the build (rather than just -j which is unbounded).

On the raw HTTPS vs gRPC topic, one thing I want to also investigate is adding proxy support (including auth to the proxy). Now that I know the agent is open source it's much easier to do! :) However, I'm not sure which option is better to support proxies.

[lustefaniak]

@steveniemitz Hey Steven, do you have your changes published in some public repo?

[steveniemitz]

looks like I never pushed it, I will tomorrow when I get to work :)