Deadlock in glib between fork and exec #774
Comments
|
I think you are correct. Note that MSan fork() interceptor
does StackDepotLockAll() before fork and similar unlock after. ASan should
do the same.
I did not realize ASan uses stack depot at all.
…On Tue, Feb 28, 2017 at 4:18 PM, Eric Rahm ***@***.***> wrote:
The Firefox project is seeing occasional deadlocks
<https://bugzilla.mozilla.org/show_bug.cgi?id=1342980> when running
Firefox under ASAN on Linux. The root cause has been tracked down to glib
allocating and freeing memory between fork and exec
<https://bugzilla.gnome.org/show_bug.cgi?id=738620>. The hypothesis is
that when the process is forked an internal ASAN lock is held, the forked
process then tries to acquire the lock when allocating or freeing and
becomes deadlocked.
Example stack:
(gdb) info threads
Id Target Id Frame
- 1 Thread 0x7f9062736700 (LWP 1728) "dconf worker" 0x000000000041eb1d
in proc_yield ()
at /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/
lib/asan/../sanitizer_common/sanitizer_atomic_clang_x86.h:23
(gdb) bt
#0 0x000000000041eb1d in LockSlow() () at /builds/slave/moz-toolchain/
src/llvm/projects/compiler-rt/lib/asan/../sanitizer_common/
sanitizer_atomic_clang_x86.h:23
#1 <#1> 0x000000000041eb1d in
LockSlow() () at /builds/slave/moz-toolchain/
src/llvm/projects/compiler-rt/lib/asan/../sanitizer_common/
sanitizer_mutex.h:53
#2 <#2> 0x00000000004ca891 in
Put() () at /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/
lib/sanitizer_common/sanitizer_mutex.h:32
#3 <#3> 0x00000000004ca891 in
Put() () at /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/
lib/sanitizer_common/sanitizer_mutex.h:179
#4 <#4> 0x00000000004ca891 in
Put() () at /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/
lib/sanitizer_common/sanitizer_persistent_allocator.h:52
#5 <#5> 0x00000000004ca891 in
Put() () at /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/
lib/sanitizer_common/sanitizer_persistent_allocator.h:67
#6 <#6> 0x00000000004ca891 in
Put() () at /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/
lib/sanitizer_common/sanitizer_stackdepotbase.h:126
#7 <#7> 0x00000000004ca16d in
StackDepotPut() () at /builds/slave/moz-toolchain/
src/llvm/projects/compiler-rt/lib/sanitizer_common/
sanitizer_stackdepot.cc:112
#8 <#8> 0x0000000000420dcc in
QuarantineChunk() () at /builds/slave/moz-toolchain/
src/llvm/projects/compiler-rt/lib/asan/asan_allocator.cc:488
#9 <#9> 0x00000000004b2c00 in
__interceptor_free() () at /builds/slave/moz-toolchain/
src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:41
#10 <#10> 0x00007f909b8d781a
in g_free ***@***.***=0x60b0002d5ee0) at /build/glib2.0-prJhLS/glib2.0-
2.48.2/./glib/gmem.c:189
#11 <#11> 0x00007f909b917cdd
in do_exec (search_path_from_envp=0, search_path=1, envp=0x0,
argv=0x604000465550, file=) at /build/glib2.0-prJhLS/glib2.0-
2.48.2/./glib/gspawn.c:1799
#12 <#12> 0x00007f909b917cdd
in do_exec (child_err_report_fd=47, stdin_fd=, stdout_fd=, stderr_fd=51,
***@***.***=0x0, ***@***.***=0x604000465550,
envp=0x0, close_descriptors=1, search_path=1, search_path_from_envp=0,
stdout_to_null=0, stderr_to_null=0, child_inherits_stdin=0,
file_and_argv_zero=0, child_setup=0x0, user_data=0x0)
at /build/glib2.0-prJhLS/glib2.0-2.48.2/./glib/gspawn.c:1229
We've worked around this
<http://searchfox.org/mozilla-central/rev/60ae6514e4c559c0c234f0e7aefccb101b8beb2e/memory/replace/logalloc/LogAlloc.cpp#82-111>
behavior in glib a few times in our own malloc hooks by registering a
pthread_atfork handler that acquires the offending lock prior to forking
and releases it once forked.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#774>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAZuSvyce-wO-Hm9rFl5ff4PsA7imM2sks5rhLlcgaJpZM4MPD-W>
.
|
|
Should be fixed for {A, M, L}San by https://reviews.llvm.org/rL304285. |
|
Looks like the fix was rolled back in https://reviews.llvm.org/rL304735 was there another fix after that? |
|
Nope, there has been no follow-up as far as I can see.
…On Sat, Dec 21, 2019 at 10:26 AM Daniil Bondarev ***@***.***> wrote:
Looks like the fix was rolled back in https://reviews.llvm.org/rL304735
was there another fix after that?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#774?email_source=notifications&email_token=AADG4SU2IOZ63PGXRJ63RUTQZZNT7A5CNFSM4DB4H6LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHPBC2I#issuecomment-568201577>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADG4SXW74INDAXXKEJKQZLQZZNT7ANCNFSM4DB4H6LA>
.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The Firefox project is seeing occasional deadlocks when running Firefox under ASAN on Linux. The root cause has been tracked down to glib allocating and freeing memory between fork and exec. The hypothesis is that when the process is forked an internal ASAN lock is held, the forked process then tries to acquire the lock when allocating or freeing and becomes deadlocked.
Example stack:
We've worked around this behavior in glib a few times in our own malloc hooks by registering a
pthread_atforkhandler that acquires the offending lock prior to forking and releases it once forked.The text was updated successfully, but these errors were encountered: