cmd/compile: archsimd VPSRLW missing constant-folding rewrite rules + suboptimal NotEqual codegen

[kolkov]

Go version

go1.26.1

Output of go env in your module/workspace:

GOEXPERIMENT=simd
GOARCH=amd64

What did you do?

I'm working on a regex engine (coregex) that uses a Teddy multi-pattern SIMD prefilter. I rewrote the core loop from hand-written Go assembly to archsimd intrinsics, hoping to get equivalent performance while also unlocking AVX2 (our asm AVX2 path was unusable due to VZEROUPPER overhead at Go/asm boundary, see #77647).

The hot loop is a standard SIMD nibble-matching pattern — load 32 bytes, extract low/high nibbles, VPSHUFB lookup, AND, check for non-zero. This loop runs ~187K iterations on a 6MB input.

Reproducer showing the two issues:

//go:build goexperiment.simd && amd64

package main

import (
	"simd/archsimd"
	"unsafe"
)

//go:noinline
func teddyLoop(data []byte, loTable, hiTable *[32]int8) int {
	nibbleMask := archsimd.BroadcastInt8x32(0x0F)
	zero := archsimd.Int8x32{}
	lo := archsimd.LoadInt8x32(loTable)
	hi := archsimd.LoadInt8x32(hiTable)

	count := 0
	for i := 0; i+32 <= len(data); i += 32 {
		chunk := archsimd.LoadInt8x32((*[32]int8)(unsafe.Pointer(&data[i])))
		loNibbles := chunk.And(nibbleMask)
		hiNibbles := chunk.AsUint16x16().ShiftAllRight(4).AsInt8x32().And(nibbleMask)

		loResult := lo.PermuteOrZeroGrouped(loNibbles)
		hiResult := hi.PermuteOrZeroGrouped(hiNibbles)
		combined := loResult.And(hiResult)

		if combined.NotEqual(zero).ToBits() != 0 {
			count++
		}
	}
	return count
}

func main() {
	data := make([]byte, 1024)
	var lo, hi [32]int8
	_ = teddyLoop(data, &lo, &hi)
}

What did you see happen?

Two codegen issues in the hot loop (go build -gcflags='-S'):

1) ShiftAllRight(4) uses register-form VPSRLW instead of immediate

The constant 4 is reloaded into a GP register and moved to XMM every iteration:

; inside the loop body:
MOVL    $4, CX          ; reload constant
MOVQ    CX, X6          ; GP → XMM
VPSRLW  X6, Y4, Y4      ; register-form shift

I dug into the SSA rules and found the root cause. The VPSRLW256const opcode exists in simdAMD64ops.go (line 1448), but the constant-folding rewrite rule is missing. Left shifts have it, right shifts don't:

; simdAMD64.rules — left shift constant folding EXISTS:
(VPSLLW256 x (MOVQconst [c])) => (VPSLLW256const [uint8(c)] x)

; right shift constant folding is MISSING:
; (VPSRLW128 x (MOVQconst [c])) => (VPSRLW128const [uint8(c)] x)  — not there
; (VPSRLW256 x (MOVQconst [c])) => (VPSRLW256const [uint8(c)] x)  — not there
; (VPSRLW512 x (MOVQconst [c])) => (VPSRLW512const [uint8(c)] x)  — not there

Same gap exists for VPSRLD and VPSRLQ non-masked variants — the const opcodes are defined but the rewrite rules to use them are absent. The masked variants (VPSRLWMasked*) have load-merging rules for the const form but also lack the basic MOVQconst folding rule.

Expected: VPSRLW $4, Y4, Y4 — one instruction.

2) NotEqual(zero).ToBits() generates 3 YMM ops instead of 1 scalar op

VPCMPEQB Y4, Y4, Y5    ; all-ones via self-compare
VPCMPEQB Y4, Y1, Y4    ; compare to zero
VPXOR    Y5, Y4, Y4    ; NOT
VPMOVMSKB Y4, DI       ; extract

This could be:

VPCMPEQB Y1, Y4, Y4    ; compare to zero
VPMOVMSKB Y4, DI       ; extract (bit=1 where zero, bit=0 where candidate)
NOTL    DI              ; scalar NOT — or just branch on the inverted condition

The self-compare + VPXOR is 2 wasted YMM operations per iteration to do what a single NOTL (or an inverted branch) could do.

What did you expect to see?

For issue 1: immediate-form VPSRLW $4, Y-reg, Y-reg. The fix should be straightforward — add the missing rules to simdAMD64.rules, matching the pattern already used for VPSLLW/VPSRAW/VPSRAD/etc.

For issue 2: strength-reduce the NOT from 2 YMM ops to 1 scalar op after VPMOVMSKB. Or recognize that NotEqual(zero).ToBits() != 0 is equivalent to Equal(zero).ToBits() != 0xFFFFFFFF and skip the inversion entirely.

Performance impact

Benchmarked on AMD EPYC 7763 (our regex-bench CI, 6MB input, 8-pattern Teddy search):

Implementation	Time	Notes
Hand-written Go asm (SSSE3, 16B/iter)	6.05 ms	baseline run
archsimd intrinsics (AVX2, 32B/iter)	34.57 ms	archsimd run, 5.7x regression
Rust regex (AVX2, inlined)	0.70 ms	same run, reference

DNA benchmark (9 Teddy patterns, same 6MB input) shows similar results: baseline vs archsimd — all patterns 34-71x slower than Rust.

archsimd AVX2 should be roughly 2x faster than SSSE3 asm (double the vector width), but the extra instructions per iteration + larger loop body appear to cause significant overhead on this microarchitecture.

Issue 1 alone adds 2 wasted instructions per iteration (the constant reload). Issue 2 adds 2 more. Over 187K iterations that's ~750K unnecessary instructions.

Related

• simd/archsimd: architecture-specific SIMD intrinsics under a GOEXPERIMENT #73787 — archsimd tracking issue. @mcy noted LICM not happening, which is related but the shift issue here is specifically a missing rewrite rule, not a general LICM problem.
• simd/archsimd: archsimd.LoadUint8x32Slice() emits nops #76968 — LoadUint8x32Slice emitting unnecessary nops (similar codegen quality area)

[gabyhelp]

Related Issues

• simd/archsimd: would intrinsics eliminate Go/assembly boundary overhead for SIMD prefilter? #77647 (closed)
• cmd/compile: on `AMD64` slow instruction sequence to Zext a < register size instruction output #76066
• cmd/asm: x86, incorrect Optab entry - PSRLW #13010 (closed)
• simd/archsimd: archsimd.LoadUint8x32Slice() emits nops #76968 (closed)
• cmd/compile: generate better SSA code for shift-by-constant #12022 (closed)
• cmd/compile: `SH(L|R)DQ` not used for things like `x<<7 | x2>>57` on amd64 #72075 (closed)
• cmd/compile: suboptimal bits.Rotate* on arm64 #48465 (closed)
• cmd/compile: missed peephole optimizations #10637 (closed)
• cmd/compile: multiplication strength-reduction rules hurting performance #21434 (closed)

Related Code Changes

• cmd/compile, simd/archsimd: add VPSRL immeidate peepholes

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[JunyangShao]

Thanks for bringing up the issue!

(VPSRLW128 x (MOVQconst [c])) => (VPSRLW128const [uint8(c)] x) is added in 1.27 dev branch.
You can checkout this CL to get them: go.dev/cl/748501.

For the NotEqual(...).ToBits() pattern, maybe we can peephole them as well. If the masks stays vectors for other uses we still need VPXOR to perform the not.

[kolkov]

@JunyangShao Thank you for the fast response!

Great to hear the VPSRLW const-folding is already in 1.27 — that was the biggest issue of the three.

For NotEqual(zero).ToBits() — in our case (Teddy SIMD prefilter), the vector result of NotEqual is never used again, only ToBits() matters. So the ideal codegen would be:

VPCMPEQB Y_combined, Y_zero, Y_tmp
VPMOVMSKB Y_tmp, R_bits
NOTL R_bits                    ; scalar NOT, 1 cycle

instead of the current:

VPCMPEQB Y_zero, Y_zero, Y_ones   ; all-ones
VPCMPEQB Y_combined, Y_zero, Y_tmp
VPXOR Y_ones, Y_tmp, Y_result     ; vector NOT
VPMOVMSKB Y_result, R_bits

A peephole for VPMOVMSKB(VPXOR(VPCMPEQB(x,x), VPCMPEQB(a,b))) → NOTL(VPMOVMSKB(VPCMPEQB(a,b))) would save 2 vector ops per loop iteration.

Also, the third issue in the report — bounds checks not being eliminated inside for i := 0; i+32 <= len(data); i += 32 loops with LoadInt8x32((*[32]int8)(unsafe.Pointer(&data[i]))) — is that a known limitation, or should I file a separate issue for BCE?

[JunyangShao]

Also, the third issue in the report — bounds checks not being eliminated inside for i := 0; i+32 <= len(data); i += 32 loops with LoadInt8x32((*[32]int8)(unsafe.Pointer(&data[i])))

Hmm that's something we had put some effort into addressing, but IIRC our solutions are incomplete.

Can you try rewriting the loop to follow this pattern: https://go.dev/play/p/auxB_-t7Z71 ?
Like use sub-slicing instead of indexing.

I will take a look into the prove and loopbce pass too.

[kolkov]

@JunyangShao Thanks for the playground link!

That sub-slicing pattern (for len(data) >= 32 { ... ; data = data[32:] }) makes sense — I'll try rewriting the loop that way. Our current code uses the index-based pattern (for i := 0; i+32 <= len(data); i += 32) with unsafe.Pointer(&data[i]), so switching to sub-slicing should help BCE reason about the bounds.

Will report back on whether the bounds checks are eliminated with this approach.