cmd/compile: simd operations not marked as hoistable

[balasanjay]

Go version

N/A

Output of go env in your module/workspace:

N/A

What did you do?

I wrote a simple base64 encoder: https://go.dev/play/p/SaXeuZMg6Xy

I used some relatively simple pure archsimd functions inside my core encoding loop (e.g. LoadUint8x64Array and BroadcastUint32x16).

What did you see happen?

For encodeSIMD1, I see a fairly messy disassembly for the inside of the loop:

Disassembly

command-line-arguments_encodeSIMD1_pc105:
        VMOVDQU64       (DI), Z1
        MOVL    $264305664, DX
        VMOVD   DX, X2
        VPBROADCASTD    X2, Z2
        MOVL    $67108928, R9
        VMOVD   R9, X3
        VPBROADCASTD    X3, Z3
        MOVL    $4129776, R10
        VMOVD   R10, X4
        VPBROADCASTD    X4, Z4
        MOVL    $16777232, R11
        VMOVD   R11, X5
        VPBROADCASTD    X5, Z5
        NOP
        LEAQ    command-line-arguments..autotmp_43(SP), R12
        MOVUPS  X15, (R12)
        MOVUPS  X15, 16(R12)
        MOVUPS  X15, 32(R12)
        MOVUPS  X15, 48(R12)
        MOVB    $1, command-line-arguments..autotmp_43(SP)
        MOVW    $258, command-line-arguments..autotmp_43+2(SP)
        MOVL    $67437316, command-line-arguments..autotmp_43+4(SP)
        MOVQ    $723682103795385863, R12
        MOVQ    R12, command-line-arguments..autotmp_43+8(SP)
        MOVQ    $1157723140823845901, R13
        MOVQ    R13, command-line-arguments..autotmp_43+16(SP)
        MOVQ    $1591764177852305939, R15
        MOVQ    R15, command-line-arguments..autotmp_43+24(SP)
        MOVQ    $2025805214880765977, R15
        MOVQ    R15, command-line-arguments..autotmp_43+32(SP)
        MOVQ    $2459846251909226015, R15
        MOVQ    R15, command-line-arguments..autotmp_43+40(SP)
        MOVQ    $2893887288937686053, R15
        MOVQ    R15, command-line-arguments..autotmp_43+48(SP)
        MOVQ    $3327928325966146091, R15
        MOVQ    R15, command-line-arguments..autotmp_43+56(SP)
        VMOVDQU64       command-line-arguments..autotmp_43(SP), Z6
        VPERMB  Z1, Z6, Z1
        VPANDD  Z1, Z2, Z2
        VPMULHUW        Z3, Z2, Z2
        VPANDD  Z1, Z4, Z1
        VPMULLW Z5, Z1, Z1
        VPORD   Z1, Z2, Z1
        VPERMB  Z0, Z1, Z1
        NOP
        NOP
        NOP
        NOP
        NOP
        VMOVDQU64       Z1, (AX)
        ADDQ    $-64, CX
        MOVQ    CX, command-line-arguments..autotmp_54+128(SP)
        NEGQ    CX
        SARQ    $63, CX
        ANDL    $64, CX
        ADDQ    CX, AX
        ADDQ    $-64, BX
        ADDQ    $-48, R8
        ADDQ    $48, DI
        ADDQ    $-48, SI
        MOVQ    command-line-arguments..autotmp_54+128(SP), CX
command-line-arguments_encodeSIMD1_pc432:
        CMPQ    SI, $64
        JLT     command-line-arguments_encodeSIMD1_pc460
        NOP
        CMPQ    BX, $63
        JHI     command-line-arguments_encodeSIMD1_pc105

For encodeSIMD2, I manually hoisted some of the magic constants out of the loop. This cleaned up the codegen a lot.

Disassembly

command-line-arguments_encodeSIMD2_pc317:
        VMOVDQU64       (DI), Z6
        VPERMB  Z6, Z5, Z6
        VPANDD  Z0, Z6, Z7
        VPMULHUW        Z1, Z7, Z7
        VPANDD  Z2, Z6, Z6
        VPMULLW Z3, Z6, Z6
        VPORD   Z6, Z7, Z6
        VPERMB  Z4, Z6, Z6
        NOP
        NOP
        VMOVDQU64       Z6, (AX)
        ADDQ    $-64, CX
        MOVQ    CX, DX
        NEGQ    DX
        SARQ    $63, DX
        ANDL    $64, DX
        ADDQ    DX, AX
        ADDQ    $-64, BX
        ADDQ    $-48, R8
        ADDQ    $48, DI
        ADDQ    $-48, SI
command-line-arguments_encodeSIMD2_pc407:
        CMPQ    SI, $64
        JLT     command-line-arguments_encodeSIMD2_pc424
        NOP
        CMPQ    BX, $63
        JHI     command-line-arguments_encodeSIMD2_pc317

What did you expect to see?

I expected to see similar codegen for encodeSIMD1 and encodeSIMD2. The compiler should hoist these operations out of the loop, I've seen it do the same for other operations, it should be possible for these SIMD operations as well.

[gabyhelp]

Related Issues

• simd/archsimd: archsimd.LoadUint8x32Slice() emits nops #76968 (closed)
• simd/archsimd: AVX instructions do not make use of memory operands #78159 (closed)
• cmd/compile: archsimd VPSRLW missing constant-folding rewrite rules + suboptimal NotEqual codegen #78138
• cmd/compile: missing inlining and/or SROA for SIMD code #79322
• simd/archsimd: missing combined load/op optimization #79408 (closed)
• cmd/compile: inefficient assembly code on arm64 #43357
• simd/archsimd: would intrinsics eliminate Go/assembly boundary overhead for SIMD prefilter? #77647 (closed)
• cmd/compile: missed optimization when using verbose array initialization #54612
• cmd/objdump: doesn't properly disassemble code generated with simd/archsimd intrinsics #78221 (closed)

Related Discussions

• Writing SIMD ASM

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[randall77]

Lifting ops out of loops is a fairly recent thing https://go-review.googlesource.com/c/go/+/697235. The set of instructions allowed was mostly set in https://go-review.googlesource.com/c/go/+/772964. SIMD ops were for the most part not allowed.

The tricky part here is we don't want to lift such operations past their cpu feature check. So we'd need to have something more complicated to allow instructions that depend on feature checks to move. Either only do it when GOAMD64 is set to a certain value, or we'd need to distinguish feature check branches vs others. (And that latter one is tricky, see this test.)

[dr2chase]

There is a feature-check pass that propagates it to blocks, that ought to help with this. See ssa/cpufeatures.go.

[balasanjay]

One interesting addendum: when I was experimenting with a vectorized base64 decoder, I ran into the following situation:

func decodeVals(archsimd.Uint8x64) archsimd.Uint8x64 {
  c0 := archsimd.LoadUint8x64Array(&[64]uint8{
		// constant table in here
	})
}

func decodeSIMD() {
  for {
    decodeVals(...)
  }
}

In other words, the constant in this case was inside a helper function; I was expecting the helper to be inlined and then the constants hoisted out of the loop.

It was straightforward enough to work around by passing in the constants into the helper function.
Compare decodeVals and decodeVals2 in https://go-review.googlesource.com/c/go/+/792880/5/src/encoding/base64/base64_simd.go#58 for the exact code.
This change made a significant performance difference (IIRC, it was a 4x increase in throughput).