Repository navigation
runtime: use SIMD for arm64 memmove (CL 692295) #80899
Description
Update 2026-09-25: the size-gate proposal and the loop-benchmark numbers below are superseded. Benchmarked on CL 692295 itself, the gate costs far more than it saves; see this comment.
Background
cmd/compile's inlined arm64 struct/array copy (OpARM64LoweredMove/LoweredMoveLoop) already uses NEON paired loads/stores (FLDPQ/FSTPQ) as of CL 692315 (merged 2025-10-15, benchmarked on a Raspberry Pi 5 / Cortex-A76). The out-of-line runtime.memmove function body (src/runtime/memmove_arm64.s, used for slice append/copy, larger struct copies the compiler can't inline, etc.) still uses only paired scalar LDP/STP for every size above 16 bytes — it never touches a V/Q register.
Prior art
CL 692295 ("runtime: optimize memmove using SIMD on arm64", Fannie Zhang, ARM) already proposes exactly this: a faithful port of the existing memmove_arm64.s algorithm — same 16-byte alignment fixup, same software-pipelined loop64, same backward/overlap path — with every LDP/STP swapped for FLDPQ/FSTPQ. It shows solid wins (~25-45%) from 128B up through the largest size tested (64KiB), benchmarked on Neoverse N2.
The CL got a thorough, cordial review from Vasiliy Leonenko (who also separately authored and landed the compiler-side CL 692315 above), including a prototyped alternative with different instruction merging that the author compared against and found didn't beat the ARM-optimized-routines-derived approach. The thread ended amicably in August 2025 with no unresolved technical objection, no Code-Review score either way, and no activity since — it appears to have stalled for lack of a maintainer decision rather than for cause. Given it's now a year stale, I'd like to help move this forward rather than let a second contributor's effort go the same way.
What we found benchmarking independently
Using an isolated micro-benchmark (same loop shape, decoupled from memmove's alignment/overlap machinery, so as not to just re-derive CL 692295's own numbers) across AWS Graviton3/4/5 (Neoverse V1/V2/V3) plus an Ampere Altra (Neoverse N1), with benchstat-verified significance (p=0.001 throughout, n=7):
- NEON wins cleanly on all four cores from 128B up to ~32KiB, no exceptions.
- Above ~32–48KiB (roughly L1D capacity on the V-series cores), V1/V2/V3 flip to a small (~1.5–2%) scalar advantage that persists out to at least 1MiB. N1 does not flip — it keeps favoring NEON at every size tested, consistent with CL 692295's own N2 data showing wins persisting to 64KiB (N-series cores appear not to hit the same crossover V-series cores do).
- There's also a small, reproducible NEON regression specific to Neoverse V3 in the 256–512B range (up to +8.9%) that doesn't show up on V1/V2/N1 — a real hardware quirk, not benchmark noise (confirmed identically across two independent harness versions).
This suggests CL 692295's unconditional "NEON for everything above 128B" is very likely a net win everywhere it's been tested, but may be leaving a small amount of performance on the table on newer V-series hardware at large sizes where it hasn't been tested — and that a size-gated variant (roughly 128B–32KiB, falling back to the existing scalar path outside that range) may be the better default if a single implementation has to serve all arm64 cores.
Proposal
Revive this work — either by getting CL 692295 unstuck as-is, or with a follow-up CL building directly on its approach (same technique, explicitly crediting Fannie Zhang's original patch and Vasiliy Leonenko's review) that adds the size gate above. Happy to put together the latter and/or offer our cross-generation benchmark data as review input on the original CL — whichever the maintainers/original author would find more useful.
Activity
lizthegrey commentedon Aug 16, 2026
cc @cherrymui @zhangfannie @vleonen — tagging you since this issue is directly about CL 692295 and its review.
To be clear about provenance: the numbers below are from our own standalone loop micro-benchmark, not a benchmark of CL 692295 itself (which we haven't run). It isolates just the register-file choice — paired scalar
LDP/STP(matchingruntime.memmove's current arm64 loop) vs. a single 4-register NEONVLD1.P/VST1.Pper 64-byte chunk — with no alignment fixup, no software pipelining, and no overlap handling, so it's a narrower test than CL 692295's own full-algorithm port. Harness, full per-size data, and rawbenchstatoutput (-count=7, p=0.001 throughout unless noted) for all four cores: https://gist.github.com/lizthegrey/5372c06fa75dd477d47f1c5899ee7364Condensed table below (negative % = NEON faster):
128B-32768B wins on all four cores, no exceptions. Past 32768B the three Graviton (V-series) cores flip to a small but consistent scalar advantage (bold, holds flat to 1MiB); the Neoverse N1 core never flips. The 256-512B regression on V3 specifically reproduced on repeated runs, so it looks like a real hardware effect rather than noise.
Happy to run this same harness against CL 692295's actual patch if that would help move the review forward.
gabyhelp commentedon Aug 16, 2026
Related Issues
Related Code Changes
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
gopherbot commentedon Aug 23, 2026
Change https://go.dev/cl/819840 mentions this issue:
runtime: use SIMD for arm64 memmove, gated to <=32768 byteslizthegrey commentedon Sep 26, 2026
Update after benchmarking CL 692295 itself. The size gate and loop-benchmark claims above don't hold up. I'd like to see CL 692295 land as is, and I've abandoned my CL 819840.
BenchmarkMemmove*Memmove/1024MemmoveOverlap/64MemmoveOverlap/64regression is real, but the alternatives I tried to fix it cost more elsewhere.Method: c6g/c7g/c8g/c9g.xlarge, 16 fresh interleaved processes per variant; two rounds agreed within ~1 point.
[-]runtime: use NEON for arm64 memmove in the ~128B-32KiB range (building on CL 692295)[/-][+]runtime: use SIMD for arm64 memmove (CL 692295)[/+]gopherbot commentedon Sep 28, 2026
Change https://go.dev/cl/840906 mentions this issue:
runtime: optimize memmove using SIMD on arm64gopherbot commentedon Sep 28, 2026
Change https://go.dev/cl/840905 mentions this issue:
runtime: use named registers in arm64 memmovelizthegrey commentedon Sep 28, 2026
Split into two CLs, one define refactoring-only, one that adds semantic changes, per @randall77.