Stack overflow in trap handlers

[zuevmaxim]

Why do we need a call to orig_trapHandler here

async-profiler/src/profiler.cpp

Line 838 in a237cad

orig_trapHandler(signo, siginfo, ucontext);

?

As I can see trapHandler method is only called from AllocTracer::trapHandler which is installed as orig_trapHandler. I think that it leads to a crash:
crash2050223224811971188.txt.

We use a patched version of the async-profiler built from master d29cb86 (I can share dylib in DM)

[apangin]

Can you reproduce the issue with unmodified async-profiler?

setupSignalHandlers() sets SIGTRAP handler to AllocTracer::trapHandler while storing the previous value in orig_trapHandler. The stack trace from the crash log suggests that setupSignalHandlers() has been called twice. I don't see how this may happen in the original async-profiler code.

An obvious workaround is to check that orig_trapHandler is not equal to (void*)AllocTracer::trapHandler. But I don't want to apply it without knowing the root cause of the issue (why setupSignalHandlers() was called twice).

[zuevmaxim]

We don't call setupSignalHandlers() more than async-profiler does. Can it be possibly called from Hooks::init?

[apangin]

We don't call setupSignalHandlers() more than async-profiler does

The problem might be related to anything - even to the way how the libary is compiled/linked. So I kindly ask you to see if the issue occurs with the genuine async-profiler or provide more evidence that it's a bug in the original code.

Can it be possibly called from Hooks::init?

Hooks::init is intended for native applications; it is not normally called in a JVM process.

[zuevmaxim]

I see setupSignalHandlers method is called twice with my dylib. Once from VM init, and once from Hooks::init (that is called from dlopen). Can I comment out DLLEXPORT methods from hooks.cpp if I'm not interested in native applications profiling?

[apangin]

Thank you for the update. DLLEXPORT functions from hooks.cpp should not be called, unless libasyncProfiler is preloaded using LD_PRELOAD on Linux or DYLD_INSERT_LIBRARIES on macOS.
It's OK to comment them out in your branch, but I'm still wondering why they are called in your case. Can you check if the error happens with the binary from here? https://github.com/async-profiler/async-profiler/files/12647822/async-profiler-3.0-ea-macos.zip

[zuevmaxim]

Thank you for your help!
The crash happened only a cople of times on some anonymous user machine, so I can't check whether it is reproducible with the original agent

[apangin]

I refactored hooks and setupSignalHandlers in recent commits.
The mentioned issue should not occur anymore.