tcmalloc deadlock

[alk]

Originally reported on Google Code with ID 611

on CentOS 6.5.
application startup deadlock.

backtrack:
#0  0x00007ffff7da8d30 in sys_futex (w=0x7ffff7fc5240, value=2, loop=<value optimized
out>) at ./src/base/linux_sysc
#1  base::internal::SpinLockDelay (w=0x7ffff7fc5240, value=2, loop=<value optimized
out>) at ./src/base/spinlock_lin
#2  0x00007ffff7daab7c in SpinLock::SlowLock (this=0x7ffff7fc5240) at src/base/spinlock.cc:133
#3  0x00007ffff7d9e493 in Lock () at src/base/spinlock.h:71
#4  SpinLockHolder () at src/base/spinlock.h:136
#5  tcmalloc::ThreadCache::InitModule () at src/thread_cache.cc:314
#6  0x00007ffff7d8dbcd in tcmalloc::ThreadCache::GetCache () at src/thread_cache.h:420
#7  0x00007ffff7dac855 in do_malloc_no_errno (size=56) at src/tcmalloc.cc:1112
#8  do_malloc (size=56) at src/tcmalloc.cc:1119
#9  do_malloc_or_cpp_alloc (size=56) at src/tcmalloc.cc:1039
#10 tc_malloc (size=56) at src/tcmalloc.cc:1578
#11 0x0000003dd740cbad in _dl_map_object_deps () from /lib64/ld-linux-x86-64.so.2
#12 0x0000003dd7412aa1 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#13 0x0000003dd740e1b6 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#14 0x0000003dd74124fa in _dl_open () from /lib64/ld-linux-x86-64.so.2
#15 0x0000003dd7d26e10 in do_dlopen () from /lib64/libc.so.6
#16 0x0000003dd740e1b6 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#17 0x0000003dd7d26f67 in __libc_dlopen_mode () from /lib64/libc.so.6
#18 0x0000003dd7cfeb25 in init () from /lib64/libc.so.6
#19 0x0000003dd800cd33 in pthread_once () from /lib64/libpthread.so.0
#20 0x0000003dd7cfec54 in backtrace () from /lib64/libc.so.6
#21 0x00007ffff7dab030 in GetStackTrace_generic (result=0x625010, max_depth=30, skip_count=3)
at src/stacktrace_gene
#22 0x00007ffff7daac0e in GetStackTrace (result=<value optimized out>, max_depth=<value
optimized out>, skip_count=<
#23 0x00007ffff7d9b65a in RecordGrowth (this=0x665000, n=<value optimized out>) at
src/page_heap.cc:584
#24 tcmalloc::PageHeap::GrowHeap (this=0x665000, n=<value optimized out>) at src/page_heap.cc:610
#25 0x00007ffff7d9bd5b in tcmalloc::PageHeap::New (this=0x665000, n=2) at src/page_heap.cc:156
#26 0x00007ffff7d99cd7 in tcmalloc::CentralFreeList::Populate (this=0x7ffff7fc6680)
at src/central_freelist.cc:329
#27 0x00007ffff7d99f00 in tcmalloc::CentralFreeList::FetchFromOneSpansSafe (this=0x7ffff7fc6680,
N=1, start=0x7fffff
#28 0x00007ffff7d99fa3 in tcmalloc::CentralFreeList::RemoveRange (this=0x7ffff7fc6680,
start=0x7fffffffe438, end=0x7
#29 0x00007ffff7d9e045 in tcmalloc::ThreadCache::FetchFromCentralCache (this=<value
optimized out>, cl=<value optimi
#30 0x00007ffff7dac860 in do_malloc_no_errno (size=1) at src/tcmalloc.cc:1112
#31 do_malloc (size=1) at src/tcmalloc.cc:1119
#32 do_malloc_or_cpp_alloc (size=1) at src/tcmalloc.cc:1039
#33 tc_malloc (size=1) at src/tcmalloc.cc:1578
#34 0x00007ffff7d8d36f in TCMallocGuard::TCMallocGuard (this=<value optimized out>)
at src/tcmalloc.cc:921
#35 0x00007ffff7da5a23 in __static_initialization_and_destruction_0 () at src/heap-profiler.cc:598

Reported by cppcoffee on 2014-03-20 01:13:54

[alk]

Yes, that's known problem at this time.

GNU/Libc based backtrace() stacktrace capturing is causing deadlocks.

Please use libunwind until this is fixed.

Reported by alkondratenko on 2014-03-20 01:20:59

[alk]

My work here: https://github.com/alk/gperftools/tree/wip-emergency-malloc already passes
unit tests and should work on GNU/Linux.

Some other OSes as well as more testing is still pending.

Reported by alkondratenko on 2014-04-02 04:11:12

[alk]

Still observed deadlocks while running tcmalloc tests on mips64 platform.

Using latest gperftools-2.2 sources.

Reported by jogarao.nartu on 2014-04-22 14:30:42

[alk]

And also observed stacktrace compilation issues with latest gperftools-2.2 on mips64
platform 

Reported by jogarao.nartu on 2014-04-22 14:34:50

[alk]

Please open separate ticket(s) for mips64. With details please.

And don't expect anything beyond --enable-minimal (just malloc) to work on mips at
this time.

Reported by alkondratenko on 2014-04-22 18:54:03

[alk]

This is now fixed by merging of emergency malloc