Skip to content

runtime: amortize zeroing and its publication for small-object spans #81393

Open

Description

@amusman

Problem

Currently allocations that need zeroed memory (this includes all from
scannable spans and some from noscan spans) pay the cost per object — a
small memclr and a publication barrier after it, repeated for every
slot of a span.

For noscan objects the publication barrier only matters to the program:
no pointer bits are recorded for them, so the GC never scans their
contents. The barrier merely ensures that code that observes the
noscan object through the allocated pointer sees zeros rather than
stale memory. A noscan allocation that does not need zeroing
(mallocgc with needzero=false) therefore has nothing to order;
ordering belongs to the caller (see #77074). And zeroing such
allocations in advance is pure waste.

For scannable objects the barrier needs to stay per-object to order
both the contents and the heap bits, but the (always required) zeroing
amortizes just the same.

Proposal

When an allocation with needzero=true triggers a refill of a span whose
live objects form a dense prefix (allocCount == freeindex; every empty
span qualifies), zero the span's free object slots in one bulk operation
and mark the span pre-zeroed (span.needzero = 0), so every subsequent
allocation from that span skips its per-object zeroing:

  • refill zeroes the free slots [freeindex*elemsize, s.limit) when
    the span is not already pre-zeroed; one publication barrier after the
    clear orders it for the whole span.
  • For needzero=false refill does nothing: the span may keep serving
    allocations that don't need zeroing, and any that do are zeroed
    individually, as today (span.needzero stays 1); once recycled, a
    needzero=true refill can still bulk-zero it.
  • Tiny and noscan allocations gate both the zeroing and its publication
    barrier on span.needzero != 0.

Safety rests on the existing invariant freeindex <= allocCount:

  • at refill every slot below freeindex is allocated, so
    allocCount == freeindex implies no live object at or above it.
  • sweep resets freeindex to 0 and allocCount to the live count, and
    sets span.needzero=1 whenever it frees an object.

Experiment results

Microbenchmarks, geomean vs base (n=10, GOMAXPROCS=1), by benchmark family (MakeSlice includes its Copy variants, AppendSlice includes AppendSliceLarge):

machine Malloc MakeSlice GrowSlice AppendSlice MixedNeedzero
Orange Pi (CIX A720) −17.6% −20.1% −4.9% −1.0% −29.2%
Kunpeng 920 −24.5% −6.6% −5.4% −0.4% −13.4%
Xeon E5-2690 v3 +0.1% −1.1% −1.4% ~0% −1.0%

On Kunpeng 920 all 158 Mallocgc subbenchmarks win.
On the Xeon per-object zeroing is cheap and the publication barrier is a no-op, leaving little to amortize.

Threading needzero into refill prevents 18–35% regressions on large AppendSlice benchmarks, the same kind of waste CL 359477 removed from mheap.alloc.

sweet benchmarks (markdown, gopher-lua, bleve-index, esbuild, etcd, tile38) and go-build:

machine sweet geomean go-build geomean
Orange Pi (CIX, A720) −1.31% ~0%
Kunpeng 920 −1.97% −0.38%
Xeon E5-2690 v3 −0.53% −0.49%

Activity

  1. gopherbot commented on Sep 8, 2026

    @gopherbot
    Contributor

    Change https://go.dev/cl/829148 mentions this issue: runtime: amortize zeroing and its publication for small-object spans

  2. JunyangShao commented on Sep 11, 2026

    @JunyangShao
    Contributor

    @golang/runtime

  3. added
    NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.
    on Sep 11, 2026
  4. amusman commented on Sep 16, 2026

    @amusman
    ContributorAuthor

    Adding benchstat output here to clean up it in review
    rt_kp920.txt
    rt_x86.txt
    sweet_kp920.txt
    sweet_opi.txt
    sweet_x86.txt

  5. mrkfrmn commented on Sep 22, 2026

    @mrkfrmn
    Contributor
  6. amusman commented on Sep 24, 2026

    @amusman
    ContributorAuthor

    I benchmarked four variants: base, only the zeroing amortization (barriers untoughed), only the change moving noscan barriers under the zeroing condition and both changes together (as in CL):
    benchstat4_rt_opi.txt
    benchstat4_sweet_kp920.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.compiler/runtimeIssues related to the Go compiler and/or runtime.

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions