mmap heap profiling deadlocks

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Run program to profile
LD_PRELOAD=/opt/google-perftools/lib/libtcmalloc.so.0  
HEAPPROFILE=/root/myprog.hprof HEAP_PROFILE_MMAP=true 
HEAP_PROFILE_MMAP_LOG=true myprog
2. It writes to output and locks.
Starting tracking the heap
mmap(start=0x0, len=69632, prot=0x3, flags=0x22, fd=-1, offset=0x0) = 
0x2aaab08b8000

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?
google-perftools-0.91 compiled from sources with gcc-3.2 on FC6 x86_64 
arch.

Please provide any additional information below.
The first deadlock appeared when heap profiler calls HeapProfilerStart 
from HeapProfilerInit with initialized mmap hooks and heap_lock held.
After I moved mmap hooks initialization to the end of HeapProfilerInit:
  HeapProfileTable::CleanupOldProfiles(fname);

  HeapProfilerStart(fname);
  if (FLAGS_mmap_profile || FLAGS_mmap_log) {
    MallocHook::SetMmapHook(MmapHook);
    MallocHook::SetMunmapHook(MunmapHook);
  }
Second deadlock appeared with following call stack (some functions seems 
to be inlined, though I compiled all programs with -g and without any -O 
flags):
#0  0x00000034da98f7a2 in __nanosleep_nocancel () 
from /lib64/tls/libc.so.6
#1  0x00002aaaaaafd57a in 
HeapLeakCheckerGlobalPrePost::~HeapLeakCheckerGlobalPrePost () at 
src/malloc_hook.cc:294
#2  0x00002aaaaaadc263 in SpinLock::Lock (this=0x2aaaaad2dac0) at 
VZLCounter.h:129
#3  0x00002aaaaaae7dc2 in RecordAlloc (ptr=0x2aaab08b8000, bytes=69632, 
skip_count=0) at src/heap-profiler.cc:195
#4  0x00002aaaaaae803a in MmapHook (result=0x2aaab08b8000, start=0x0, 
size=69632, prot=3, flags=34, fd=-1, offset=0)
    at src/heap-profiler.cc:267
#5  0x00002aaaaaae208d in MallocHook::InvokeMmapHook 
(result=0x2aaab08b8000, start=0x0, size=69632, protection=3, flags=34,
    fd=-1, offset=0) at src/google/malloc_hook.h:100
#6  0x00002aaaaaafe141 in virtual thunk to 
VZL::VZLForkableRefCounter::~VZLForkableRefCounter$delete() () at 
VZLCounter.h:129
#7  0x00002aaaaaafe5e0 in LowLevelAlloc::Alloc (request=65560, 
arena=0x2aaaadf30020) at VZLCounter.h:129
#8  0x00002aaaaaae7bb8 in ProfilerMalloc (bytes=65560) at 
src/heap-profiler.cc:110
#9  0x00002aaaaaaea5c9 in MallocHook::SetDeleteHook () at 
src/heap-checker.cc:367
#10 0x00002aaaaaaea319 in MallocHook::SetDeleteHook () at 
src/heap-checker.cc:367
#11 0x00002aaaaaae9cd0 in MallocHook::SetDeleteHook () 
at /usr/include/c++/3.2.3/bits/basic_string.h:955
#12 0x00002aaaaaae913a in MallocHook::SetDeleteHook () 
at /usr/include/c++/3.2.3/bits/basic_string.h:955
#13 0x00002aaaaaae7de8 in RecordAlloc (ptr=0x1c7021d8, bytes=1, 
skip_count=0) at src/heap-profiler.cc:197
#14 0x00002aaaaaae7f4f in NewHook (ptr=0x1c7021d8, size=1) at 
src/heap-profiler.cc:240
#15 0x00002aaaaaae0e11 in MallocHook::InvokeNewHook (p=0x1c7021d8, s=1) 
at src/google/malloc_hook.h:62
#16 0x00002aaaaaafd8be in malloc (size=1) at src/tcmalloc.cc:2785
#17 0x00002aaaaaae1323 in TCMallocGuard (this=0x2aaaaad2d860) at 
src/tcmalloc.cc:2510
#18 0x00002aaaaaae00cc in __static_initialization_and_destruction_0 
(__initialize_p=1, __priority=65535) at src/tcmalloc.cc:2525
#19 0x00002aaaaaae07f9 in global constructors keyed to 
_ZN61FLAG__namespace_do_not_use_directly_use_DECLARE_int64_instead31FLAGS_tcmall
oc_sample_parameterE 
() at src/tcmalloc.cc:572
#20 0x00002aaaaaafd886 in 
HeapLeakCheckerGlobalPrePost::~HeapLeakCheckerGlobalPrePost () at 
src/malloc_hook.cc:294

I suppose RecordAlloc should be reentrant or there should be some flag to 
suspend tcmalloc's allocation logging.

Original issue reported on code.google.com by ytimen...@gmail.com on 3 May 2007 at 8:46

[GoogleCodeExporter]

We've been able to reproduce the problem and are thinking about solutions.  I 
expect
we'll have a fix in our next release.

Original comment by csilv...@gmail.com on 7 May 2007 at 3:36

• Changed state: Accepted

[GoogleCodeExporter]

A status update on this, since the fix didn't make it into the 0.92 release.  
We have
a fix coded up, but there's some debate going on over it.  I hope we'll have a
resolution by the next release after this.

Original comment by csilv...@gmail.com on 19 Jul 2007 at 2:42

[GoogleCodeExporter]

Another status update, sadness: it's not in the 0.93 release either.  It turned 
out
to be a pretty major change. It's near the end of the review process now, I 
hope, so
maybe in 0.94?

Original comment by csilv...@gmail.com on 17 Aug 2007 at 9:28

[GoogleCodeExporter]

Another update.  The review process did not yield a consensus for the change to 
go in
-- it added a lot of complexity, and we were having a hard time understanding 
the
resulting code.  In the end, we felt it was better to keep the code as-is, with 
this
bug, than to put in a fix that affects maintainability so much.

That said, I've decided to attach the patch below.  For those who really need 
the
functionality, feel free to apply this patch.  We're pretty sure it works 
correctly,
but of course there's even less warranty for this patch than usually.  We also 
can't
guarantee it will continue to apply cleanly as we upgrade google-perftools.

I've verified that the patch applies cleanly against perftools 0.94-rc1 (not
released), and should apply to perftools 0.94 when it comes out.

Original comment by csilv...@gmail.com on 26 Nov 2007 at 9:53

• Changed state: WontFix

Attachments:

• p

[GoogleCodeExporter]

It only took the good part of a year, but we finally found some time to fix this
properly!  The fix should be out in the next release of perftools.

Original comment by csilv...@gmail.com on 2 Apr 2008 at 3:35

• Changed state: Started

[GoogleCodeExporter]

Fixed (finally!) in google-perftools 0.97

Original comment by csilv...@gmail.com on 22 Apr 2008 at 1:50

• Changed state: Fixed