runtime/pprof: memory profiles appear to undercount allocations #10537
Description
Apologies in advance that this is vague.
I was looking recently into compiler allocations, and because I wanted to know the breakdown by type, I hacked up tracealloc and friends in mprof.go to print exactly one line per allocation and nothing else (including suppressing output in tracefree and tracegc). Surprisingly, this output differs significantly from that reported by the memory profiler, even at runtime.MemProfileRate=1.
For example, running go tool 6g -memprofile=mem.prof html/template/*.go and then running go tool pprof -alloc_objects yields a count of 458,794 allocations, almost none of which are from cmd/internal/gc.walkexpr.
Setting runtime.MemProfileRate=1 and repeating the experiment turns up 607,429 allocations, 5.87% of which are from cmd/internal/gc.walkexpr. (The np **Node parameter escapes, causing an allocation of a *Node, which is only 8 bytes, thus the previous significant undercounting.)
Running with GODEBUG=allocfreetrace=1 go tool 6g html/template/* 2>&1 | grep "tracealloc" | wc -l yields 1,032,392 allocations, which is 70% more than reported by the memory profiler! This number is very stable across repeated runs, with variation less than 0.01%. Only logging allocs with size > 16 in tracealloc yields a count of 740,816 allocations, so this is probably not a tiny alloc issue.
There are probably multiple things to do here:
- Understand the mismatch and fix the memory profiler.
- Add a memprofilerate flag to the compiler.
- Warn in pprof when
-alloc_objectsis used with memprofilerate != 1, because allocation counts can be dramatically skewed by the size of the allocation.