Repository navigation
runtime: amortize zeroing and its publication for small-object spans #81393
Description
Problem
Currently allocations that need zeroed memory (this includes all from
scannable spans and some from noscan spans) pay the cost per object — a
small memclr and a publication barrier after it, repeated for every
slot of a span.
For noscan objects the publication barrier only matters to the program:
no pointer bits are recorded for them, so the GC never scans their
contents. The barrier merely ensures that code that observes the
noscan object through the allocated pointer sees zeros rather than
stale memory. A noscan allocation that does not need zeroing
(mallocgc with needzero=false) therefore has nothing to order;
ordering belongs to the caller (see #77074). And zeroing such
allocations in advance is pure waste.
For scannable objects the barrier needs to stay per-object to order
both the contents and the heap bits, but the (always required) zeroing
amortizes just the same.
Proposal
When an allocation with needzero=true triggers a refill of a span whose
live objects form a dense prefix (allocCount == freeindex; every empty
span qualifies), zero the span's free object slots in one bulk operation
and mark the span pre-zeroed (span.needzero = 0), so every subsequent
allocation from that span skips its per-object zeroing:
- refill zeroes the free slots
[freeindex*elemsize, s.limit)when
the span is not already pre-zeroed; one publication barrier after the
clear orders it for the whole span. - For needzero=false refill does nothing: the span may keep serving
allocations that don't need zeroing, and any that do are zeroed
individually, as today (span.needzerostays 1); once recycled, a
needzero=true refill can still bulk-zero it. - Tiny and noscan allocations gate both the zeroing and its publication
barrier onspan.needzero != 0.
Safety rests on the existing invariant freeindex <= allocCount:
- at refill every slot below
freeindexis allocated, so
allocCount == freeindeximplies no live object at or above it. - sweep resets
freeindexto 0 andallocCountto the live count, and
setsspan.needzero=1whenever it frees an object.
Experiment results
Microbenchmarks, geomean vs base (n=10, GOMAXPROCS=1), by benchmark family (MakeSlice includes its Copy variants, AppendSlice includes AppendSliceLarge):
| machine | Malloc | MakeSlice | GrowSlice | AppendSlice | MixedNeedzero |
|---|---|---|---|---|---|
| Orange Pi (CIX A720) | −17.6% | −20.1% | −4.9% | −1.0% | −29.2% |
| Kunpeng 920 | −24.5% | −6.6% | −5.4% | −0.4% | −13.4% |
| Xeon E5-2690 v3 | +0.1% | −1.1% | −1.4% | ~0% | −1.0% |
On Kunpeng 920 all 158 Mallocgc subbenchmarks win.
On the Xeon per-object zeroing is cheap and the publication barrier is a no-op, leaving little to amortize.
Threading needzero into refill prevents 18–35% regressions on large AppendSlice benchmarks, the same kind of waste CL 359477 removed from mheap.alloc.
sweet benchmarks (markdown, gopher-lua, bleve-index, esbuild, etcd, tile38) and go-build:
| machine | sweet geomean | go-build geomean |
|---|---|---|
| Orange Pi (CIX, A720) | −1.31% | ~0% |
| Kunpeng 920 | −1.97% | −0.38% |
| Xeon E5-2690 v3 | −0.53% | −0.49% |
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsTodo
Activity
gopherbot commentedon Sep 8, 2026
Change https://go.dev/cl/829148 mentions this issue:
runtime: amortize zeroing and its publication for small-object spansgabyhelp commentedon Sep 8, 2026
Related Issues
Related Code Changes
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
JunyangShao commentedon Sep 11, 2026
@golang/runtime
amusman commentedon Sep 16, 2026
Adding benchstat output here to clean up it in review
rt_kp920.txt
rt_x86.txt
sweet_kp920.txt
sweet_opi.txt
sweet_x86.txt
mrkfrmn commentedon Sep 22, 2026
cc @matloob
amusman commentedon Sep 24, 2026
I benchmarked four variants: base, only the zeroing amortization (barriers untoughed), only the change moving noscan barriers under the zeroing condition and both changes together (as in CL):
benchstat4_rt_opi.txt
benchstat4_sweet_kp920.txt