I tried to use pprof to debug a memory leak in a long running program that was showing growth in live objects (MemStats.Mallocs - MemStats.Frees) over time.
What did you expect to see?
pprof to reveal the problematic code that was causing the leak.
What did you see instead?
Setting sample_index=inuse_objects and running top showed lots of objects coming from runtime.systemstack. pprof was not giving useful info regarding what is actually running on the stack and causing the leak of live objects. web and peek showed that runtime.mstart was calling runtime.systemstack suggesting a possible leak of threads.
However this was not actually relevant, I think due to this line in asm_arm.s that states "make it look like mstart called systemstack on g0, to stop traceback".
After lots of manual debugging, the culprit was found to be a bug in my code where a defer was being called in a hot and long running loop, rather than being called once before the loop.
@josharian encouraged me to file an issue after helping me debug this over on gophers #performance slack. I have solved my memory leak bug but pprof didn't really help in this case - I thought it might be useful to share my experience and start a discussion on whether pprof could be improved to catch issues like this in the future.