proposal: simd/archsimd: enable AMD64 architecture-specific SIMD by default

[cherrymui]

Proposal Details

In Go 1.26, we introduced AMD64 architecture-specific SIMD intrinsics in the simd/archsimd package under GOEXPERIMENT=simd.

Thanks to users trying out the experiment, we've received valuable feedback. The feedback is very positive, and we did not find major flaws in the API. While there are opportunities for further optimizations and the addition of new features, these improvements can be implemented without necessitating revisions to the existing API.

The architecture-specific SIMD API is the first step and the lower level of our Two-level approach. It serves as the building block for the upper-level, portable, size-agnostic vector API. We now have sketched out a draft of the high-level API. Our analysis confirms that the simd/archsimd package is compatible with the high-level API, provides a good foundation, and facilitates efficient interoperability (see #78902 for details).

Further, CL 745080 demonstrates that this approach can be adapted well to WebAssembly (128-bit fixed-size vectors), and the ongoing discussions shows that a similar, but scalable, API is emerging to support scalable vectors with ARM64 SVE. This suggests that the current approach can work reasonably well on a variety of architectures.

Based on these findings, we propose to enable the AMD64 simd/archsimd API by default, without requiring GOEXPERIMENT=simd.

API changes

So far the only API we'd want to change from Go 1.26 experimental API is to change ShiftAll{Left,Right} operations to take an uint8 instead of uint64. This is to make it consistent with the RotateAll{Left,Right} and ShiftAll{Left,Right}Concat operations. (CL 767560)

API additions

Based on the discussions and feedback on #73787, we plan to add the following APIs to AMD64 simd/archsimd.

• More operations on masks: bit operations AndNot, Not, Xor; testing operations All, Any, None, which check if all/any/no elements are selected; and the String method.

func (MaskMxN) AndNot(MaskMxN) MaskMxN
func (MaskMxN) Not() MaskMxN
func (MaskMxN) Xor(MaskMxN) MaskMxN
func (MaskMxN) All() bool
func (MaskMxN) Any() bool
func (MaskMxN) None() bool
func (MaskMxN) String() string

• Broadcasting a vector to a wider vector: BoradcastAllTo{4,8,...}

func (Int64x2) BroadcastAllTo4() Int64x4
func (Int64x2) BroadcastAllTo8() Int64x8
func (Int64x4) BroadcastAllTo8() Int64x8
func (Int32x4) BroadcastAllTo8() Int32x8
func (Int32x4) BroadcastAllTo16() Int32x16
func (Int32x8) BroadcastAllTo16() Int32x16

and also for Uint and Float elements.

• Quadruple dot products: DotProductQuadruple{Signed,Unsigned} (discussion)

func (Int8x16) DotProductQuadrupleSigned(Int8x16) Int32x4
func (Int8x16) DotProductQuadrupleUnsigned(Uint8x16) Int32x4
func (Uint8x16) DotProductQuadrupleSigned(Int8x16) Int32x4
func (Uint8x16) DotProductQuadrupleUnsigned(Uint8x16) Uint32x4

• Convenient methods (see below)

Convenient methods

We plan to add a few more convenient methods, which may not necessarily map to a single machine instruction, but still simple, usually a small number of instructions.

During the development of the high-level, portable API, we found that there are some operations on one architecture that can be easily implemented on another. For example, on WebAssembly, there is an instruction that negates the integer elements of a vector. On AMD64, there is no such an instruction, but it can be done with a Sub from a zero vector. Adding this convenient method makes it easy to support the operation in the portable layer. And it can also be handy to users.

We currently already have some convenient methods, for instance, Not is Xor with an all-one vector, and some comparison operations are a logical combination of other, native comparison operations. We plan to extend this list with Neg on integer vectors.

func (IntMxN) Neg() IntMxN

API summary

A compact summary of the API is at #73787 (comment). The full list of the current experimental API can be found at https://pkg.go.dev/simd/archsimd@master. The new API will be the experimental API with the adjustments mentioned above.

Related/future work

• #76175: a static checker of SIMD CPU features. This ensures that SIMD API is used only when the corresponding CPU feature is supported, preventing illegal instruction crashes.
• #78902: A proposal for high-level portable SIMD API
• CL 745080: An implementation of 128-bit simd/archsimd API for Wasm
• Ongoing discussion about the simd/archsimd API for ARM64 SVE

[gabyhelp]

Related Issues

• simd/archsimd: architecture-specific SIMD intrinsics under a GOEXPERIMENT #73787
• proposal: simd: architecture and vector-size agnostic SIMD intrinsics under a GOEXPERIMENT #78902
• simd: rename low-level SIMD package to simd/archsimd (GOEXPERIMENT) #76473 (closed)
• proposal: simd/m128: single instruction multiple data API #60149 (closed)
• proposal: add package for using SIMD instructions #53171 (closed)
• proposal: simd: new package for intrinsics #67520 (closed)
• simd/archsimd, ppc64le: track SIMD (GOEXPERIMENT=simd) enablement for PowerPC64LE #78338

Related Code Changes

• cmd/compile, internal/simd: generated codes for SIMD

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[eliasnaur]

Exposing archsimd before the portable high-level simd package seems backwards to me. I see archsimd as an escape hatch for when simd is insufficient, which means that archsimd should only expose what can't be efficiently expressed in simd.

Assuming you're moving forward with exporting archsimd, I consider a reasonable way of portably testing archsimd code essential. For example, I'd like to test amd64 SIMD on my arm64 machine, even after Apple removes Rosetta 2 emulation.

You list #76175 as "related/future" work. I consider a checker essential so architecture-specific code doesn't leak to package exported API and to encourage portable fallbacks.

[aclements]

Proposal review discussion:

This is a big (albeit fairly shallow) API. Ways we've validated it:

• A good number of people hammered on the GOEXPERIMENT version in Go 1.26.
• The simd/archsimd/internal/simd_test package has test coverage against software emulations of a large fraction of the API.
• We have a prototype of WASM support, which is similarly fixed-width, and a lot of work on ARM64 SVE support, which is variable width and thus quite different. This helps validate cross-platform compatibility (one name means one thing). The only "mistake" these turned up in the AMD64 API was the uint64 argument to ShiftAll{Left,Right}.
• We have a prototype of the (admittedly small for now) portable layer built atop archsimd. The approach is to essentially take the intersection of operations supported across the platforms, adding thin emulations to the archsimd packages where necessary to fill out that intersection. The prototype covers all sizes of AMD64 (128, 256, 512), ARM64 NEON, wasm, and a pure software emulation, and it has tests shared across all of the platforms to ensure they're consistent.

To @eliasnaur 's points: all good points. The choice to expose archsimd before simd is purely pragmatic. We had to build archsimd first in order to build simd. 🤷

We would love to have a "pure software" emulation so you can test code on platforms you don't have, and it's something the SIMD group has thought about. It's just more work. The simd package actually has some of the foundation for this in its tests.

I plan to return to the feature checker (#76175) shortly after the freeze. That's not in the main repo anyway, so it's not tied to the toolchain release cycle. I have a prototype and a lot of the tricky foundations are already in x/tools.

[aclements]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— aclements for the proposal review group

[aclements]

So far the only API we'd want to change from Go 1.26 experimental API is to change ShiftAll{Left,Right} operations to take an uint8 instead of uint64. This is to make it consistent with the RotateAll{Left,Right} and ShiftAll{Left,Right}Concat operations.

Minor nit: I believe the shift operations implement "Go semantics", so overshifts result in all zeros (or all ones). That suggests that the shift count should not be uint8 for any of these, but rather uint or possibly uint64. The danger with uint8 is that code is unlikely to have a variable shift count already in a uint8, and the natural temptation is to satisfy the API by just doing uint8(count). But if count is > 255, you lose out on the "Go semantics" and get some unpredictable result. Given the way we handle overshifting, these ops should just take a large type for the shift count.

For the rotates it doesn't matter as much because they're modular anyway, but they should be consistent with shifts.

[aclements]

@cherrymui, I think we need to remove ClearAVXUpperBits or maybe leave it gated behind the GOEXPERIMENT if we really need it for now. It's not generally safe and its behavior is impossible to describe at a language level.

[eliasnaur]

To @eliasnaur 's points: all good points. The choice to expose archsimd before simd is purely pragmatic. We had to build archsimd first in order to build simd. 🤷

Sure, but this proposal is about exporting archsimd without GOEXPERIMENT, locking its wide waist API down forever. Once locked down, we risk a go/types vs the Go compiler internal types2 situation, where the portable simd package needs to reimplement some parts of archsimd. This is a bigger risk because SIMD is all about performance and even minor inefficiencies in archsimd's API will be tempting to reimplement. We also risk conversion between simd and archsimd types for occasional access to architecture specific instructions ending up less efficient than with a holistic design.

For concreteness: I suggest keeping archsimd under GOEXPERIMENT until there is a (GOEXPERIMENT or not) version of the portable simd package available for users. I also suggest we wait until at least one vector-based SIMD instruction set (ARM SVE or RISC-V's counterpart) is available in archsimd and as a backend for simd.

The downsides for users are the need to set GOEXPERIMENT and the risk of API changes in archsimd. However, any such API change will very likely be done to support simd which is exactly what we want. Further, once simd is available I expect a bunch of archsimd code should be rewritten to use simd anyway (I'd even go as far and unexport all API from archsimd that have simd counterparts of similar efficiency).

[cherrymui]

ClearAVXUpperBits is not an unsafe operation. It is totally safe semantically. In particular, it does not clobber any live variables (we save/restore them if any). Performance aside, it is semantically no-op.

That said, it may be good to put ClearAVXUpperBits behind the experiment. I think the compiler can just insert it automatically, one day. On the other hand, people who need it today still need to enable the experiment, which partly diminishes the point of enabling the API by default. And enabling the experiment may also bring other things like #78902, which may or may not be desirable.

[aclements]

I did my best to cram the entire API into my head and review it, with the help of an audit tool that I'll post shortly.

Pre-GA

Major

Load/store (CL, CL)

• Load<T>, Load<T>Slice, Load<T>SlicePart(s []<bN>) T - Since the higher-level API is entirely scalable, should we swap the "default" here? That is, make this Load<T>(s []<bN>) T, Load<T>Array(s y *[M]<bN>) T, and Load<T>Part(s []<bN>) T. Likewise with Store. In this case, Load<T>Masked should probably be renamed, too.
• LoadMasked<T>(y *[M]<bN>, mask Mask) T - What's the scalable equivalent of this operation? (Might relate to my next question.)
• LoadMasked<T>(y *[M]<bN>, mask Mask) T - This takes an array, but sometimes the point of masking is to avoid reading past the end of memory. Should this take a slice instead and check if the mask would violate the slice bounds? Or just say this is what Load*SlicePart is for?

Likely violations of "one name one meaning" portability principal:

• CarrylessMultiply - The control arguments do not seem portable (or friendly), despite the generic name. More specific name for the operation with control arguments? CarrylessMultiplyEven without the controls? Replace control arguments with EvenOdd, OddEven, Odd variants? Even and Odd variants map to PMULLB and PMULLT on SVE, supporting portability and scalability of those names (ARM does not support mixing hi/lo). (CL)

"As" methods:

• I'm skeptical of the As* "reinterpration" methods in general. It's a ton of API surface (270 methods!) and scales O(n^2), despite being pretty obscure. It doesn't carry its weight. It's also way too temptingly named; people will reach for them expecting to get a numeric conversion. My recommendation:

  • For IntNxL <=> UintNxL (same width, same lanes), provide ConvertToX. These follow Go's convertion rules anyway.
  • To convert floats to and from bits, provide FloatNxL.Bits() UintNxL and UintNxL.FloatFromBits() FloatNxL, following the convensions of the math package.
  • To interpret the element width (and the number of lanes), provide very explicit UintNxL.BitcastToUint<Nₒ>() Uint<Nₒ>x<Lₒ> methods (only on uint vectors).

  This is much more explicit and the "scaryness" of the names is far more proportional to the "scaryness" of the operation. It achieves an O(n) surface by separating element reinterpretation from lane reinterpretation.

Medium

• Broadcast1To<N> - Should we drop these and make them be a peephole of archsimd.Broadcast<T>(x.GetElem(0))? These are only supported on 128 bit wide vector types anyway, which is exactly what GetElem is supported on. Short term: just drop, add peephole later. (Unexport CL, no peephole yet)

• T.SumAbsDiff(y T) <BN*2>x<L/2> - Since this groups every 8 uint8 elements into a uint64 value, this seems much easier to think about if it returns a Uint64x<L/8>. (CL)

• T.ConvertToInt32(), T.ConvertToInt64(), T.ConvertToUint32(), T.ConvertToUint64() - These are all truncating conversions, but there are also rounding conversions. I'm guessing we called these "Convert" because they match the behavior of Go float->int, but maybe we want to be explicit and call these all TruncToX. Those will be in good company with the other Trunc(ate)ToX methods. We can separately introduce RoundToX (CVTP[DS]2(DQ|PI)) later.

• ClearAVXUpperBits is a wart. Keep it behind the GOEXPERIMENT for now?

Shift/rotate

• T.ConcatShiftBytesRight(Grouped)?(shift uint8, y T) T, T.ShiftAll(Left|Right)Concat(shift uint8, y T) T - Swap the arguments so it matches "concat(x,y)>>shift" and matches T.Shift(Left|Right)Concat(y T, shift T) T
• T.ShiftLeft(y T) T and T.ShiftRight(y T) T - The shift amounts should be unsigned vectors. (CL)
• T.Shift(All)?(Left|Right)Concat(shift uint8, y T) T - Unlike all of the other Shift* methods, which behave like Go shifts, these use (shift % element width). What's extra weird is that these don't even match their non-Concat equivalents. What's extra extra weird is that they're mod the element width, NOT (2 * element width). Options:
  • Drop them. They're horrible. (But also useful.)
  • Add ModN at the end of the name (where N is the element width)
  • Give them logical, Go-like semantics. ShiftAll* is free to fix. The variable form is harder.

Pair/Concat mixups - For the most part, Pair refers to adjacent pairs of elements (e.g., AddPairs, DotProductPairs) and Concat refers to first logically concatenating two vectors and then operating on that concatenated vector (e.g., ConcatPermute). But we're not fully consistent about this.

• T.AddPairs(y T) T and AddPairs* are concatenated and paired operations. Rename to ConcatAddPairs? This is true of all AddPairs* methods and SubPairs* methods.
• SelectFromPair* and Select128FromPair* are concatenated operations. I think these are misnamed in other ways because they're just PermuteScalars with concatenation. We already have ConcatPermute. I suggest ConcatPermuteScalars* and ConcatPermute128Scalars*. This also indicates the argument order should put y first, and then the indexes. That would match ConcatPermute and also seems to make more sense.

Minor

• T.ToMask() (to Mask) - There are at least two things this could mean. x86 people might expect this to make the mask from the sign bits, especially on Ints. But it's actually "x != 0". Maybe we should rename it to T.NotEqualZero() Mask (allowing T.EqualZero) or T.NonZeros() Mask (allowing T.Zeros)? It's tempting to call it IsNonZero, but IsZero() bool already exists and returns the conventional bool rather than a Mask. We could separately introduce T.SignToMask() Mask with the x86 semantics (vpmovq2m, etc). Leave as is (we can add SignToMask/SignMask later)

• Mask.ToInt8x16() - Given that this sets all bits in the result, why is this ToInt instead of ToUint? Should we have both? Leave as is (we can add ToUint, but also ToInt.ToBits does this)

• T.Merge(y T, mask Mask) T - This keeps x if mask is true, y if mask is false. Isn't that backwards of what you'd expect?

• T.AddSub(y T) T - Rename to AddOddSubEven. Also make MulAddSub and MulSubAdd into MulAddOddSubEven and MulAddEvenSubOdd. These are a mouthful, but so are the operations.

• T.CopySign does not copy the sign, and it's not the same operation as math.Copysign. It multiplies by the sign of y, so should probably be named MulSign. (CL)

• T.MulHigh(y T) T - The "High" in this refers to keeping the high part of the result, while elsewhere we use Hi to refer to the upper half of an input vector. This might be confusing, though I don't have any better suggestions. Leave it.

• T.SumAbsDiff(y T) <BN*2>x<L/2> - The name doesn't mention the 8-way grouping. Rename to SumOf8AbsDiff

Conversions

• T.TruncateToX() T - The "same type" rounding method is called T.Trunc. Let's rename these to TruncToX. (Also the math/big rounding mode is called Trunc.)
• T.RoundToEven() T - We recently accepted short rounding mode names in math/big and they match the SIMD methods except for this one. Let's rename this to Round to match. "Round to even" is the default rounding mode in IEEE 754, so "Round" implies "to even". Same with other RoundToEven* methods.

Shifts/rotates

• All *Shift* and *Rotate* methods should take uint64. Currently a few take uint8.
• T.Shift(Left|Right)Concat(y T, shift T) T - Unlike all of the "Concat*" methods, this "concatenates" each individual element. Should we use a different verb? We do put it at the end, which helps.
• T.ShiftAllRightConcat(shift uint8, y T) T - The order of x and y seems backwards. The element-wise concatenation is currently y ++ x. In contrast, ShiftAllLeftConcat does x ++ y. On the other hand, with the current behavior, x.ShiftAllRightConcat(s, zero) == x.ShiftAllRight(s).

Not urgent (can be post-GA)

Major

A lot of the documentation is simultaneously a bit wordy and a bit vague because it tries to do a lost in prose. I wonder if we should establish a psuedo-code convention like z[i] = x[i] * y[i] for making the docs more precise. Toying with a few examples:

• Broadcast1To16 copies the lowest element of its input to all 16 elements of the output vector.

  =>

  Broadcast1To16 copies x[0] to all output elements. z[i] = x[0]

• ConcatPermute performs a full permutation of vector x, y using indices:

  result = {xy[indices[0]], xy[indices[1]], ..., xy[indices[n]]}
  

  where xy is the concatenation of x (lower half) and y (upper half).
  Only the needed bits to represent xy's index are used in indices' elements.

  =>

  ConcatPermute permutes the concatenation of x and y. Out-of-range indices wrap around.

  w := concat(x, y)
  z[i] = w[indices[i] % len(w)]
  

• AddPairsGrouped horizontally adds adjacent pairs of elements. With each
  128-bit as a group: for x = [x0, x1, x2, x3, ...] and y = [y0, y1, y2, y3,
  ...], the result is [x0+x1, x2+x3, ..., y0+y1, y2+y3, ...].

  =>

  AddPairsGrouped splits x and y into 128-bit groups, and within each group, concatenates x and y and horizontally adds adjacent pairs of elements.

  for each group xg, yg, zg {
    w := concat(xg, yg)
    zg[i] = w[i*2] + w[i*2+1]
  }
  

Minor

Bad argument names:

• T.AESRoundKeyGenAssist(rconVal uint8) T
• T.SHA1FourRounds(constant uint8, y T) T
• Constant Shift and Rotate methods should all use the same argument name and type.

Docs could be improved:

• AddPairsGrouped and SubPairsGrouped
• CarrylessMultiply

Trivial extensions:

• T.ToMask - Why is this only defined on Ints and not Uints? (If we extend this to NonZeros it would make sense on floats, too.)
• T.InterleaveHi - Trivial to extend to Float32xN and Float64xN.

Slightly non-trivial extensions:

• T.Abs() T - Weirdly, not defined on floats. This can be efficiently emulated with Broadcast and AndNot with some type punning.
• T.Ceil(), T.Floor(), T.RoundToEven(), T.Trunc() - Missing from 512-bit vectors, but can easily be emulated with T.XScaled(0).

Weird support matrix:

• LoadMasked<T>(y *[M]<bN>, mask Mask) T and T.StoreMasked(y *[M]<bN>, mask Mask)
• Add*Saturated

We have all of the Go bitwise operations except "complement" (^x).

T.GaloisFieldAffineTransform and T.GaloisFieldAffineTransformInverse - I'm not sure what we do with a non-constant b today, but that can be emulated quite efficiently. (To the point where we could leave b out of the API and peephole x.GFAT(y).Xor(BroadcastUint8(b))).)

Not a problem, but I had to think about it

T.DotProductPairs(y T) <BN*2>x<L/2> - I was worried that the layout of this might be non-portable, but I think it's reasonable. It does not have an SVE equivalent: the closest is SDOT, but SDOT has frankly weird layout so I'm okay with it having a more obscure name.

T.IsZero() bool returns a bool for the whole vector, but IsNaN returns a Mask. We should stick with IsZero() bool because that's a common convention. I don't think the divergence is too confusing.

T.ShiftAll(Left|Right)Concat(shift uint8, y T) T - Should "Concat" go at the front? No! This doesn't actually concatenate x and y before shifting, so it's not like other "Concat*" operations.