testing: allocs reported incorrectly depending on the --benchtime and --count flags

[ainar-g]

Go version

go version go1.25.1 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/ainar/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/ainar/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build4247421775=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/home/ainar/agrd/urlfilter/go.mod'
GOMODCACHE='/home/ainar/go/pkg/mod'
GONOPROXY='REDACTED'
GONOSUMDB='REDACTED'
GOOS='linux'
GOPATH='/home/ainar/go'
GOPRIVATE='REDACTED'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/ainar/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.1.linux-amd64'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/ainar/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='go1.25.1'
GOTOOLDIR='/home/ainar/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.1.linux-amd64/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25.1'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

1. Checkout github.com/AdguardTeam/urlfilter v0.21.0.

2. go test --bench='BenchmarkSeqScanTable_AppendMatching_baseFilter' --run='^$' ./internal/lookup/
   go test --bench='BenchmarkSeqScanTable_AppendMatching_baseFilter' --benchtime=10s --run='^$' ./internal/lookup/
   go test --bench='BenchmarkSeqScanTable_AppendMatching_baseFilter' --benchtime=1s --count=10 --run='^$' ./internal/lookup/

What did you see happen?

allocs/op goes between 4 (the first and third call) and 0 (the second call). B/op is also drastically different between them.

If I add --memprofile=mem.out and inspect mem.out with go tool pprof, it point to the code in the benchmark setup. That is, the code that goes before b.ReportAllocs and b.Loop calls, for example loadTable.

What did you expect to see?

Consistent results which only include the allocations from the for b.Loop() body.

[ainar-g]

For reference, this is the go tool pprof output mentioned:

ROUTINE ======================== github.com/AdguardTeam/urlfilter/internal/lookup_test.BenchmarkSeqScanTable_AppendMatching_baseFilter in /home/ainar/agrd/urlfilter/internal/lookup/seqscantable_test.go
    6.32MB    76.18MB (flat, cum) 85.09% of Total
         .          .     82:func BenchmarkSeqScanTable_AppendMatching_baseFilter(b *testing.B) {
    6.32MB     6.32MB     83:   s := newStorage(b, string(baseFilterData))
         .          .     84:   tbl := &lookup.SeqScanTable{}
         .    69.86MB     85:   loadTable(b, tbl, s)
         .          .     86:
         .          .     87:   r := rules.NewRequest(testURLStrBaseFilterDomain, testURLStrBaseFilterDomain, rules.TypeOther)
         .          .     88:
         .          .     89:   gotRules := make([]*rules.NetworkRule, 0, 1)
         .          .     90:

Note that b.ReportAllocs is called on line 91 and for b.Loop() { is on line 92.

[ainar-g]

Apologies for the noise. I was under the impression that benchmarks with b.ReportAllocs discard the first run of the loop, but that's not the case, and I was confusing b.ReportAllocs with testing.AllocsPerRun.

To anyone reading this, if your benchmarks need some kind of “warmup”, to e.g. allocate a slice or fill pools, you should perform that warmup explicitly.

[gabyhelp]

Related Issues

• testing: Wrong number of allocations #54041 (closed)
• cmd/pprof: discrepency between pprof list and weblist #27473 (closed)
• cmd/compile: go statement desugaring results in additional allocations #46349 (closed)
• testing: don't truncate allocs/op #24631 (closed)
• cmd/pprof: list not accurate in 1.9 #21697 (closed)
• net/http/pprof: Negative allocation counts produced by /debug/pprof/allocs?seconds=1 #49171 (closed)
• x/perf/cmd/benchstat: wrong title shown for `benchstat` when comparing memory #58025 (closed)
• runtime: pprof list command not showing allocation in 1.9beta2 #21022 (closed)
• testing: benchmark cleanup causes inconsistent and slower benchmark time #38314 (closed)
• cmd/compile: poor escape analysis of `strings.SplitSeq` #73524

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}