-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ThreadSanitizer: deadlock in signal handler #1540
Comments
This part is problematic:
What signal is it? |
I'm not sure, but probably it's SIGPROF from our ThreadFuzzer (it randomly send SIGPROF to some threads and sleeps or calls sched_yield in the signal handler). I doubt it's SIGSEGV or SIGBUS, because we attach gdb to the server process before running stress tests and I don't see such signals in the gdb output. Also there are a lot of threads with |
For SIGPROF sighandler wouldn't call CallUserSignalHandler:
|
I'm not familiar with llvm/tsan codebase, but looks like it may call CallUserSignalHandler even for SIGPROF while executing a blocking function (i.e. if And according to the stacktrace it was in a blocking function:
And looks like it have set
Binary: link Am I missing something? |
Also we noticed this deadlock after upgrading from clang-13 to clang-14. As I see from git blame, |
You are right. This is a bug around |
Here is a test and additional CHECKs that detect this situation more reliably: |
Sent https://reviews.llvm.org/D127845 with a fix. |
Hi, has the fix been merged? We faced the same issue again (this time with clang-15). I did not see the changes from https://reviews.llvm.org/D127845 in main branch in https://github.com/llvm/llvm-project, so seems like the patch was forgotten. |
Wait, what happened here? Wasn't this merged as: dvyukov/llvm-project@fdb73f7 |
That's my repo fork. |
Ah, me not looking the repo - and Github also closed this issue based on that commit... |
Since initial issue got resolved [1]. [1]: google/sanitizers#1540 Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
We set in_blocking_func around some blocking C functions so that we don't delay signal infinitely (if in_blocking_func is set we deliver signals synchronously). However, pthread_join is blocking but also call munmap/free to free thread resources. If we are inside the munmap/free interceptors called from pthread_join and deliver a signal synchronously, it can lead to deadlocks and crashes since we re-enter runtime and try to lock the same mutexes or use the same per-thread data structures. If we re-enter runtime via an interceptor when in_blocking_func is set, temporary reset in_blocking_func around the interceptor and restore it back when we return from the recursive interceptor. Also move in_blocking_func from ThreadSignalContext to ThreadContext so that we can CHECK that it's not set in SlotLocker ctor. Fixes google/sanitizers#1540 Reviewed By: melver Differential Revision: https://reviews.llvm.org/D127845
Hi, seems like our stress tests have found a deadlock in TSan while testing ClickHouse. The stacktrace is
I looked through llvm code, and looks like this thread is trying to lock the same mutex second time:
in
__tsan::CallUserSignalHandler
: https://github.com/llvm/llvm-project/blob/12e137ab24dae51553433e00ff96e28a14d5b1f5/compiler-rt/lib/tsan/rtl/tsan_mman.cpp#L272and then in
__tsan::Acquire
: https://github.com/llvm/llvm-project/blob/f831d6fc800ccf22c1c09888fce3e3c8ebc2c992/compiler-rt/lib/tsan/rtl/tsan_rtl_mutex.cpp#L448If I understand correctly,
SlotLocker
locksthr->slot->mtx
which is__sanitizer::Mutex
and it's not recursive.The binary was built with Clang 14.0.5 (build log).
The text was updated successfully, but these errors were encountered: