runtime: use NEON for arm64 memmove in the ~128B-32KiB range (building on CL 692295)

[lizthegrey]

Background

cmd/compile's inlined arm64 struct/array copy (OpARM64LoweredMove/LoweredMoveLoop) already uses NEON paired loads/stores (FLDPQ/FSTPQ) as of CL 692315 (merged 2025-10-15, benchmarked on a Raspberry Pi 5 / Cortex-A76). The out-of-line runtime.memmove function body (src/runtime/memmove_arm64.s, used for slice append/copy, larger struct copies the compiler can't inline, etc.) still uses only paired scalar LDP/STP for every size above 16 bytes — it never touches a V/Q register.

Prior art

CL 692295 ("runtime: optimize memmove using SIMD on arm64", Fannie Zhang, ARM) already proposes exactly this: a faithful port of the existing memmove_arm64.s algorithm — same 16-byte alignment fixup, same software-pipelined loop64, same backward/overlap path — with every LDP/STP swapped for FLDPQ/FSTPQ. It shows solid wins (~25-45%) from 128B up through the largest size tested (64KiB), benchmarked on Neoverse N2.

The CL got a thorough, cordial review from Vasiliy Leonenko (who also separately authored and landed the compiler-side CL 692315 above), including a prototyped alternative with different instruction merging that the author compared against and found didn't beat the ARM-optimized-routines-derived approach. The thread ended amicably in August 2025 with no unresolved technical objection, no Code-Review score either way, and no activity since — it appears to have stalled for lack of a maintainer decision rather than for cause. Given it's now a year stale, I'd like to help move this forward rather than let a second contributor's effort go the same way.

What we found benchmarking independently

Using an isolated micro-benchmark (same loop shape, decoupled from memmove's alignment/overlap machinery, so as not to just re-derive CL 692295's own numbers) across AWS Graviton3/4/5 (Neoverse V1/V2/V3) plus an Ampere Altra (Neoverse N1), with benchstat-verified significance (p=0.001 throughout, n=7):

• NEON wins cleanly on all four cores from 128B up to ~32KiB, no exceptions.
• Above ~32–48KiB (roughly L1D capacity on the V-series cores), V1/V2/V3 flip to a small (~1.5–2%) scalar advantage that persists out to at least 1MiB. N1 does not flip — it keeps favoring NEON at every size tested, consistent with CL 692295's own N2 data showing wins persisting to 64KiB (N-series cores appear not to hit the same crossover V-series cores do).
• There's also a small, reproducible NEON regression specific to Neoverse V3 in the 256–512B range (up to +8.9%) that doesn't show up on V1/V2/N1 — a real hardware quirk, not benchmark noise (confirmed identically across two independent harness versions).

This suggests CL 692295's unconditional "NEON for everything above 128B" is very likely a net win everywhere it's been tested, but may be leaving a small amount of performance on the table on newer V-series hardware at large sizes where it hasn't been tested — and that a size-gated variant (roughly 128B–32KiB, falling back to the existing scalar path outside that range) may be the better default if a single implementation has to serve all arm64 cores.

Proposal

Revive this work — either by getting CL 692295 unstuck as-is, or with a follow-up CL building directly on its approach (same technique, explicitly crediting Fannie Zhang's original patch and Vasiliy Leonenko's review) that adds the size gate above. Happy to put together the latter and/or offer our cross-generation benchmark data as review input on the original CL — whichever the maintainers/original author would find more useful.

[lizthegrey]

cc @cherrymui @zhangfannie @vleonen — tagging you since this issue is directly about CL 692295 and its review.

To be clear about provenance: the numbers below are from our own standalone loop micro-benchmark, not a benchmark of CL 692295 itself (which we haven't run). It isolates just the register-file choice — paired scalar LDP/STP (matching runtime.memmove's current arm64 loop) vs. a single 4-register NEON VLD1.P/VST1.P per 64-byte chunk — with no alignment fixup, no software pipelining, and no overlap handling, so it's a narrower test than CL 692295's own full-algorithm port. Harness, full per-size data, and raw benchstat output (-count=7, p=0.001 throughout unless noted) for all four cores: https://gist.github.com/lizthegrey/5372c06fa75dd477d47f1c5899ee7364

Condensed table below (negative % = NEON faster):

size	Graviton3 (Neoverse V1)	Graviton4 (V2)	Graviton5 (V3)	Ampere Altra (N1)
128B	-19.2%	-11.8%	-4.7%	-2.8%
256B	-21.7%	-14.0%	+8.9%	-7.4%
512B	-29.5%	-8.9%	+3.5%	-5.3%
1024B	-36.7%	-17.7%	-4.5%	-4.0%
2048B	-41.5%	-20.8%	-3.7%	-2.1%
4096B	-44.8%	-24.4%	-1.9%	-1.0%
8192B	-30.0%	-3.7%	-2.9%	-13.2%
16384B	-29.9%	-4.1%	-3.4%	-14.1%
32768B	-29.8%	-3.6%	-2.0%	-1.4%
49152B	-0.7%	+1.5%	+1.6%	-1.4%
131072B	+0.1%	+1.7%	+1.5%	-1.4%
1048576B	+0.6%	+2.2%	+1.8%	-1.4%

128B-32768B wins on all four cores, no exceptions. Past 32768B the three Graviton (V-series) cores flip to a small but consistent scalar advantage (bold, holds flat to 1MiB); the Neoverse N1 core never flips. The 256-512B regression on V3 specifically reproduced on repeated runs, so it looks like a real hardware effect rather than noise.

Happy to run this same harness against CL 692295's actual patch if that would help move the review forward.

[gabyhelp]

Related Issues

• runtime: memmove performance on arm64 for unaligned copies is poor on some CPUs #40324 (closed)
• runtime: improve arm64 memmove #22925 (closed)

Related Code Changes

• cmd/compile: use arm64 neon in LoweredMemmove/LoweredMemmoveLoop
• runtime: improve memmove performance on arm64
• runtime: improve arm64 memmove implementation
• runtime: simplify and optimize memmove for amd64
• runtime: enhance memmove() function using SVE instructions
• runtime: optimize memmove using SIMD on arm64
• runtime: adjust the arm64 memmove and memclr to operate by word as much as they can
• runtime: optimize the function memmove using SIMD on loong64

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[gopherbot]

Change https://go.dev/cl/819840 mentions this issue: runtime: use SIMD for arm64 memmove, gated to <=32768 bytes