crypto/cipher: AES CTR optimizations

[starius]

There are some optimizations which can be applied to AES CTR implementation.

• amd64: When the low 64-bit counter does not overflow within an 8-block group, take the fast path that keeps all subsequent counters in XMM registers. Fill the first counter exactly as before, then expand it in XMM space and only fall back to the scalar path if a carry is detected. This netted up to an 18% improvement for AES-128 on long buffers. (CL 714361)

• amd64: Use VAES/AVX2 to process two 128-bit blocks per instruction. VAES is already available on AMD Zen 3 and newer, where a similar optimization reportedly delivers up to a 74% gain.

• amd64: Extend VAES support to AVX-512 so we can go through four blocks at a time on wide vector machines. In Linux kernel this optimization brought up to 150% speed improvement.

• amd64: Server CPUs expose 32 vector registers. We should be able to keep the expanded key hot in registers instead of reloading it from memory every eight blocks. Not tested.

• arm64: Likewise, ARM64 gives us 32 vector registers. We can cache the expanded key in registers while the loop runs, rather than re-fetching it every iteration. Tested. Brings up to 20% speedup.

• arm64: Replace the eight scalar VLD1/VST1 pairs with wider LD1 {Vx, Vy} and ST1 {Vx, Vy} instructions (or the new LD1R forms) so we move 32 bytes per issue. Not tested.

[gopherbot]

Change https://go.dev/cl/714361 mentions this issue: crypto/internal/fips140/aes: optimize amd64

[gabyhelp]

Related Issues

• crypto/aes: speedup CTR mode on AMD64 and ARM64 #53503
• crypto/aes: AES cipher creation and encrypt / decrypt operations can be sped up significantly #65507 (closed)
• crypto/cipher: easy optimisations to xorBytes #53023
• crypto/cipher: Speed up CTR cipher #6741 (closed)
• proposal: ctr-aes support parallel xor, affected/package: crypto/cipher #59248 (closed)
• crypto/aes: implement ctrAble when using AES-NI on amd64 #20967 (closed)

Related Code Changes

• crypto/aes: speedup CTR mode on AMD64 and ARM64
• crypto/aes: add optimized implementation of AES-CTR for AMD64
• crypto/aes: Implement new and improved AES-GCM ciphers optimized with AVX-512 VAES and VPCLMULQDQ
• crypto/aes: Implement AES-GCM ciphers optimized with AVX-512 VAES and VPCLMULQDQ

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[Jorropo]

Use VAES/AVX2 to process two 128-bit blocks per instruction.

Shouldn't this be 256bits ?
I know some VAES implementations are double pumped 128bits but we would reduce the instruction dispatch overhead.

[starius]

@Jorropo IIUC, in VAES-AVX2 mode, a single instruction looks like this:

VAESENC Y1, Y0, Y0

It processes a single YMM register (i.e. 256 bits) which represents 2 AES blocks (128 bits each).

I made a small Go tool which runs this instruction to test if the CPU is capable of doing it:
https://gist.github.com/starius/0a6f84c64de0650ebeaf4381b25c2c42

---

There is also VAES-AVX512 mode, were a single VAESENC instruction is applied to a ZMM register (512 bits, 4 AES blocks), but it requires AVX512 support, which is rare compared to VAES-AVX2. VAES-AVX2 is present on regular hardware produced a couple of years ago. VAES-AVX512 is not present on any of my hardware including servers. But it is available for some instances in the cloud, for instance C3 and C3D in Google Cloud's Compute Engine.

[Jorropo]

Oh I missred what you said, yes we agree sorry.

[mknyszek]

CC @golang/security