proposal: cmd/go: allow -cpuprofile in tests with multiple packages

[abitrolly]

Proposal Details

Sometimes there are functions that make all tests slower by a fraction of second, but when scaled to hundreds of tests, these functions introduce significant delay.

While running all tests in Go 1.24 it is impossible to detect such death by a thousand cuts functions with standard -cpuprofile:

$ go test -cpuprofile cpu.out ./...
cannot use -cpuprofile flag with multiple packages

The solution is a custom script that scrapes directory recursively for package names, runs each package tests separately with -cpuprofile, then merges the results, and shows the sorted (flame)graph. But it would be easier, if go test could use -cpuprofile with multiple packages natively.

Existing solutions

gotestsum proposes the script to get JSON output and cpu.profiles in package dirs.

#!/usr/bin/env bash
set -eu

for pkg in $(go list "$@"); do
    dir="$(go list -f '{{ .Dir }}' $pkg)"
    go test -json -cpuprofile="$dir/cpu.profile" "$pkg"
done

go pprof can merge profiles.

go tool pprof -proto a.pprof b.pprof > merged.pprof

Summing it up. Create .pprof file for each package test and then group them into tests.profile.

#!/bin/bash
# script doesn't fail on errors

TESTLIST=${*:-./...}

for pkg in $(go list "$TESTLIST"); do
    filename=$(echo "$pkg" | tr "/" "-").pprof
    go test -cpuprofile="$filename" "$pkg"
done

go tool pprof -proto *.pprof > tests.profile

Outcome

Not really what is expected from a few hundreds tests. Just 1.37s of sampling from 65.86s of running time. Need to increase sampling rate to 100% somehow.

✗ go tool pprof -top tests.profile
File: api.test
Build ID: 14abffb7729557b92543923a5e4f900691cc3f60
Type: cpu
Time: 2025-02-27 21:53:49 +03
Duration: 65.86s, Total samples = 1.37s ( 2.08%)
Showing nodes accounting for 1.37s, 100% of 1.37s total
      flat  flat%   sum%        cum   cum%
     0.43s 31.39% 31.39%      0.43s 31.39%  internal/runtime/syscall.Syscall6
     0.09s  6.57% 37.96%      0.09s  6.57%  runtime.rtsigprocmask
     0.07s  5.11% 43.07%      0.07s  5.11%  runtime.futex
...

Increase test loop count

It is impossible to increase sampling rate for tests from command line, but it is possible to run test multiple times to collect more profile data.

$ go test -run . -cpuprofile cpu.out ./commands/ci/get && ls -la cpu.out
ok      gitlab.com/gitlab-org/cli/commands/ci/get       0.256s
-rw-r--r--. 1 anatoli anatoli 451 Mar  4 19:49 cpu.out
$ go test -run . -count 100 -cpuprofile cpu.out ./commands/ci/get && ls -la cpu.out
ok      gitlab.com/gitlab-org/cli/commands/ci/get       0.432s
-rw-r--r--. 1 anatoli anatoli 8301 Mar  4 19:49 cpu.out

The profiling script with loops.

#!/bin/bash
# script doesn't fail on errors

TESTLIST=${*:-./...}
LOOPS=100

for pkg in $(go list "$TESTLIST"); do
    filename=$(echo "$pkg" | tr "/" "-").pprof
    go test -cpuprofile="$filename" -count "$LOOPS" "$pkg"
done

go tool pprof -proto *.pprof > tests.profile

[gabyhelp]

Related Issues

• cmd/pprof: Feature: profiling support for regular tests #23811 (closed)
• cmd/go: make -fooprofile flags imply -c #4127 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}