proposal: runtime/pprof,runtime: new goroutine leak profile

[VladSaioc]

Proposal Details

Design document.

This proposal involves a change to the garbage collector that allows it detect some types of goroutine leaks (a.k.a. partial deadlocks in the academic community).

The approach uses memory reachability as computed by the garbage collector to single out goroutines blocked over concurrency primitives that cannot be accessed by any (transitively) runnable goroutines. It is already described at length in this publication.

The approach has been validated experimentally in several contexts. It was able to discover goroutine leaks:

• caused by the misuse of synchronization primitives (channels, sync primitives, and mixed) in 69 microbenchmarks derived from real-world concurrency bugs,
• in 3111 test suites within Uber, for libraries using concurrency features, with a rate of 180 to 357 syntactically distinct goroutine leaks when compared to uber-go/goleak (which is only applicable to unit tests)
• in a production service within Uber, where it was able to find 3 syntactically distinct goroutine leaks, for a total of 252 goroutine leak reports over 24 hours.

This proposal follows the implementation according to the available artifact, but with the following changes:

• The stacks of leaked goroutines (and other heap resources reachable only through the stacks) are not forcefully reclaimed by the GC; instead, once goroutines are flagged as leaked, they are treated as traceable by the GC, as in the runnable GC. This preserves the soundness of goroutine leak detection without changing what the GC perceives as reachable memory.
• Goroutine leak detection would be triggered on-demand by extracting a new type of profile, pprof/goroutineleak; requesting it triggers a goroutine leak detection GC cycle, followed by the collecting of a goroutine profile containing only the stacks of leaked goroutines.
• In the current implementation, changes are enabled via the goleakprofile flag for GOEXPERIMENT.

[seankhliao]

I think this would benefit from an implementation before a proposal.

[VladSaiocUber]

Implementation provided at #74622. Waiting for CLA resolution.

[randall77]

Not sure where Gaby is.
Related:
#18835
#13759

[VladSaiocUber]

@randall77 Thanks for the reference to #13759! Indeed, the proposed approach is very similar to what is proposed in this comment.

[gopherbot]

Change https://go.dev/cl/688335 mentions this issue: Proposal #74609: goroutine leak detection by using the garbage collector