Vectorize small-offset overlapping match copies with NEON tbl / SSSE3 pshufb

[thevar1able]

Summary

For small match offsets (overlap period < 16 bytes), ZSTD_wildcopy falls back to an 8-byte-per-iteration COPY8 loop whose load hits the previous iteration's store, so it runs at store-forwarding latency. This loop dominates decompression of integer and low-cardinality columnar data, whose matches frequently have offsets of 4 or 8 bytes (runs of repeated fixed-width values) — a very common shape for column-oriented storage. We hit this in ClickHouse (ClickHouse/ClickHouse#108049).

This PR materializes one full period of the repeating pattern in a vector register and stores 16 bytes per iteration, advancing the pattern with a single table-lookup instruction — tbl (vqtbl1q_u8) on AArch64, pshufb (_mm_shuffle_epi8) on x86_64 with SSSE3 — and no load inside the loop:

adv[diff][j] = (16 + j) % diff   : shift the pattern forward by 16 bytes,
                                   composes because pattern[k] = src[k % diff]

The emitted AArch64 loop is 4 instructions: tbl / str q,[x],#16 / cmp / b.hi.

Design notes:

• The new path is a standalone FORCE_NOINLINE function (ZSTD_overlapCopyShortOffset), not inlined into ZSTD_wildcopy. wildcopy is force-inlined into the sequence-execution loops many times over; an earlier version of this change expanded the vector path in place and regressed silesia.tar decompression by ~2.5% with clang on x86_64 purely through code growth in the hot decoder functions (profiling showed the new instructions themselves were cold on silesia). As a standalone function the regression disappears, and the aarch64 binary shrinks by 65 KB. The out-of-line call is cheaper than the scalar loop it replaces even at minimum trip count (see the timestamp-column result below).
• The first 16 bytes are copied with COPY8, exactly like the first two iterations of the scalar loop. This seeds the pattern register from initialized memory: a 16-byte load at ip would touch not-yet-written bytes of dst (MemorySanitizer rejects that, even though the shuffle never selects those lanes).
• Each iteration stores 16 bytes and stops once op >= oend, overrunning the logical end by at most 15 bytes — within the existing WILDCOPY_OVERLENGTH (32) slack, enforced with a ZSTD_STATIC_ASSERT.
• diff is always in [8, 16) here (callers expand smaller offsets via ZSTD_overlapCopy8 first; the scalar COPY8 loop relies on the same contract).
• The x86_64 path is gated on compile-time __SSSE3__ (baseline x86-64 builds keep the scalar loop; x86-64-v2/v3 distro builds and -march=native get the new path). The NEON path is gated on AArch64 (vqtbl1q_u8 does not exist in 32-bit ARM NEON, whose vtbl can only look up in a 64-bit table); verified that an ARMv7 NEON build compiles and falls back to scalar.
• The non-overlapping (offset >= 16) path is untouched. Compression is unaffected: the decoder's sequence execution is the only caller of ZSTD_wildcopy with ZSTD_overlap_src_before_dst. Compressed output is byte-identical with and without the patch.

Prior art

The wildcopy family has been optimized several times — #1668, #1804, #1977, #2256, and #3145 (Arm's two-stage copy) — but all of these target the non-overlapping (offset >= 16) fast path. The small-offset overlap path has remained the scalar COPY8 loop since #1804 gave it its current shape. We found no previous issue/PR proposing a shuffle-based pattern-advance for it. (LZ4 special-cases small offsets in LZ4_memcpy_using_offset, but via scalar pattern expansion.)

Benchmarks

Methodology per CONTRIBUTING.md: zstd -b# -i3+, interleaved A/B runs pinned to one core (taskset), 5-10 repetitions, best-of aggregation, warmup pass. Per-rep spread 0.06–0.92%.

x86_64 — AMD Ryzen 9 9950X3D, clang 21.1.8 / gcc 16.1.1, -O3 -mssse3

Decompression (MB/s):

input	lvl	clang dev	clang PR	ratio	gcc dev	gcc PR	ratio
silesia.tar	1	2232	2237	1.002	2359	2360	1.001
silesia.tar	3	1939	1934	0.997	2045	2062	1.008
int64, run-heavy	1	29877	35089	1.174	30291	35046	1.157
int64, run-heavy	3	25013	30876	1.234	25529	30946	1.212
int64, small deltas	1	1740	1891	1.087	1650	1841	1.116
int64, small deltas	3	1736	1892	1.090	1648	1840	1.116
int32, zipfian	1/3	—	—	1.006/0.990	—	—	1.003/1.000
int16, sparse	1/3	—	—	0.995/0.998	—	—	0.995/1.000

AArch64 — ARM Neoverse-N1 (Ampere Altra, 2.6 GHz), gcc 14.2.0, -O3

Decompression (MB/s), best-of-5:

input	lvl	dev	PR	ratio
silesia.tar	1	973.6	977.9	1.004
silesia.tar	3	780.7	781.7	1.001
int64, run-heavy	1	4230.3	12046.8	2.848
int64, run-heavy	3	4037.2	10448.3	2.588
int64, small deltas	1	848.3	868.7	1.024
int64, small deltas	3	847.9	868.0	1.024
int16, sparse	1/3	—	—	1.005/1.003
int32, zipfian	1/3	—	—	0.999/1.001

A clang 20.1.8 A/B corroborates: no silesia regression (its apparent +2-3% is uniform across control cases, i.e. a code-layout artifact, not claimed as a gain). Earlier measurements of the inline predecessor on Graviton 4 (Neoverse-V2) with real ClickBench hits columns: UserID (Int64) 2.30x, AdvEngineID (Int16) 1.46x, ClientIP (Int32) 1.42x, RegionID (Int32) 1.27x.

Correctness

• make check passes on x86_64 (-mssse3) and aarch64.
• Byte-exact round-trips at levels 1/3/12/19 over inputs engineered to produce matches at every offset 1..16 (plus 17/24/32) and over the columnar corpora, under AddressSanitizer+UBSan and MemorySanitizer, on both x86_64 and aarch64 (192/192 clean on aarch64) — including periods 8-15, exactly the 8 <= diff < 16 cases the new code handles.
• Cross-compiled libzstd.a for aarch64 (gcc) and 32-bit ARMv7 with NEON enabled (clang/zig) — the latter as a regression check for the compile-time gate.

[meta-cla]

Hi @thevar1able!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!