Repository navigation
runtime/pprof: CPU profiles incorrect for kernels with broken setitimer support #13841
Description
As of Go 1.6, pprof's CPU profiles are known to be incorrect on a few systems due to what are arguably kernel bugs. This issue documents those systems.
The text below distinguishes a profile being incomplete (missing profile samples for code that was running) from being incorrect (containing samples for code that wasn't running).
- DragonflyBSD (all known versions): Delivers a profiling signal only when a thread runs continuously for an entire clock tick (20ms). In workloads with high context-switch or garbage collection rates, this may cause profiles to be incomplete.
- Linux (without CONFIG_HIGH_RES_TIMERS=y): Delivers a profiling signal only when a thread runs continuously for an entire clock tick (often 10ms). In workloads with high context-switch or garbage collection rates, this may cause profiles to be incomplete. Most Linux kernels in use today do enable high-resolution timers and therefore do not suffer from this problem.
- NetBSD (all known versions): Delivers signals to the wrong thread. On such systems, profiles are commonly very incorrect.
- OpenBSD (all known versions): Delivers a profiling signal only when a thread runs continuously for an entire clock tick (20ms). In workloads with high context-switch or garbage collection rates, this may cause profiles to be incomplete.
- OS X (fixed in OS X 10.11 El Capitan): Deliver signals to the wrong thread. On such systems, profiles are commonly very incorrect. See rsc.io/pprof_mac_fix for a workaround on those early systems.
- Solaris (fixed in Solaris 8): Delivers a profiling signal only when a thread runs continuously for an entire clock tick (10ms). In workloads with high context-switch or garbage collection rates, this may cause profiles to be incomplete. Solaris 8 fixes the problem on systems with APIC hardware (most x86 systems). On systems that continue to exhibit the problem, adding
set hires_tick = 1to/etc/systemcan mitigate this problem somewhat by reducing the clock tick to 1ms.
Please comment on this issue only if the text above is incomplete or incorrect; we will keep this top-level comment up to date.
Activity
bradfitz commentedon Jan 6, 2016
Sent a change to skip the test on dragonfly too: https://go-review.googlesource.com/18306 after failures:
http://build.golang.org/log/9e126b69f4629739f0c7951ce4d42cc46db54a31
http://build.golang.org/log/56a3af05f787150fb668c99bfd30ebaae3e67bf8
etc
rsc commentedon Jan 6, 2016
Added Dragonfly, thanks.
aclements commentedon Jan 8, 2016
FreeBSD has the same problem as OpenBSD and DragonflyBSD, but by default runs at 1000Hz, so the problem is less noticeable.
mdempsky commentedon Jan 22, 2016
I don't believe the description of OpenBSD is entirely accurate. In particular, OpenBSD's "hard clock" interrupt runs at 100Hz (i.e., 10ms clock tick intervals) on all CPUs supported by Go. Also, I don't see any evidence that OpenBSD's kernel cares about whether the process was running for the full time slice; only that it was running when the interrupt fired. The process may only have actually run for the last 1ms of the clock slice, but it will still have its timer credited for the full 10ms.
Lastly, I'm not sure it's relevant to the runtime/pprof tests, but for completeness: OpenBSD sends SIGPROF to the process, not the thread. It's just that it favors sending to the running thread when possible. If the interrupted thread has blocked SIGPROF and the process has other threads that are not blocking SIGPROF, the kernel will send SIGPROF to one of those instead.
runtime/pprof: mark dragonfly and solaris as bad at pprof
adonovan commentedon Mar 30, 2023
Hi Russ, do you want to add #57722 to the list at the top?
gopherbot commentedon Aug 27, 2026
Change https://go.dev/cl/822704 mentions this issue:
cmd/pprof: retry TestDisasm with a longer profile duration