cmd/compile: encoding/binary.PutUint32 generates unaligned data storage on arm64 arch

[mattypiper]

What version of Go are you using (go version)?

$ go version
go version go1.20.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="auto"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/matt/Library/Caches/go-build"
GOENV="/Users/matt/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/matt/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/matt/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.20.3/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.20.3/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.20.3"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="cc"
CXX="c++"
CGO_ENABLED="1"
GOMOD=""
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/c6/500c7t352v54d_jjsr5vf2gw0000gn/T/go-build1931171172=/tmp/go-build -gno-record-gcc-switches -fno-common"
GOROOT/bin/go version: go version go1.20.3 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.20.3
uname -v: Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
ProductName:	macOS
ProductVersion:	12.6
BuildVersion:	21G115
lldb --version: lldb-1400.0.38.13
Apple Swift version 5.7.1 (swiftlang-5.7.1.135.3 clang-1400.0.29.51)
gdb --version: GNU gdb (GDB) 13.1

What did you do?

The following code generates some funky assembly on ARM64.

package main

import (
	"encoding/binary"
	"fmt"
)

func main() {
	b := make([]byte, 4)
	x := uint8(0xff)
	binary.LittleEndian.PutUint32(b, uint32(x))
	fmt.Printf("%x\n", b)
}

What did you expect to see?

I'm compiling the executables with

GOARCH=amd64 GOOS=linux go build -o putuint32_bug_amd64 putuint32_bug.go
GOARCH=arm GOOS=linux go build -o putuint32_bug_arm putuint32_bug.go
GOARCH=arm64 GOOS=linux go build -o putuint32_bug_arm64 putuint32_bug.go

On x86_64 (amd64) and arm architectures, I get reasonable results.

In amd64, we have a movl $0xff,(%rax) and movups %xmm15,0x38(%rsp) (see line 4828cd below). This correctly sets b, producing output ff000000. The instructions are concise.

$ objdump -d putuint32_bug_amd64 | grep -A25 '<main\.main>:'
00000000004828a0 <main.main>:
  4828a0:	49 3b 66 10          	cmp    0x10(%r14),%rsp
  4828a4:	0f 86 85 00 00 00    	jbe    48292f <main.main+0x8f>
  4828aa:	48 83 ec 50          	sub    $0x50,%rsp
  4828ae:	48 89 6c 24 48       	mov    %rbp,0x48(%rsp)
  4828b3:	48 8d 6c 24 48       	lea    0x48(%rsp),%rbp
  4828b8:	48 8d 05 61 7c 00 00 	lea    0x7c61(%rip),%rax        # 48a520 <type:*+0x7520>
  4828bf:	bb 04 00 00 00       	mov    $0x4,%ebx
  4828c4:	48 89 d9             	mov    %rbx,%rcx
  4828c7:	e8 f4 36 fc ff       	call   445fc0 <runtime.makeslice>
  4828cc:	90                   	nop
  4828cd:	c7 00 ff 00 00 00    	movl   $0xff,(%rax)
  4828d3:	44 0f 11 7c 24 38    	movups %xmm15,0x38(%rsp)
  4828d9:	bb 04 00 00 00       	mov    $0x4,%ebx
  4828de:	48 89 d9             	mov    %rbx,%rcx
  4828e1:	e8 da 73 f8 ff       	call   409cc0 <runtime.convTslice>
  4828e6:	48 8d 15 f3 6d 00 00 	lea    0x6df3(%rip),%rdx        # 4896e0 <type:*+0x66e0>
  4828ed:	48 89 54 24 38       	mov    %rdx,0x38(%rsp)
  4828f2:	48 89 44 24 40       	mov    %rax,0x40(%rsp)
  4828f7:	48 8b 1d 52 c6 0a 00 	mov    0xac652(%rip),%rbx        # 52ef50 <os.Stdout>
  4828fe:	48 8d 05 13 60 03 00 	lea    0x36013(%rip),%rax        # 4b8918 <go:itab.*os.File,io.Writer>
  482905:	48 8d 0d b1 7b 01 00 	lea    0x17bb1(%rip),%rcx        # 49a4bd <go:string.*+0xa5>
  48290c:	bf 03 00 00 00       	mov    $0x3,%edi
  482911:	48 8d 74 24 38       	lea    0x38(%rsp),%rsi
  482916:	41 b8 01 00 00 00    	mov    $0x1,%r8d
  48291c:	4d 89 c1             	mov    %r8,%r9

On 32-bit arm, it's a similar story. mvn r1, #0 and strb r1, [r0] sets the first byte to 0xff, followed by three remaining strb r1 instructions to offsets from r0. See line a2d18 below. This works fine.

$ objdump -d putuint32_bug_arm | grep -A25 '<main\.main>:'
000a2ce8 <main.main>:
   a2ce8:	e59a1008 	ldr	r1, [sl, #8]
   a2cec:	e15d0001 	cmp	sp, r1
   a2cf0:	9a00002b 	bls	a2da4 <main.main+0xbc>
   a2cf4:	e52de034 	str	lr, [sp, #-52]!	; 0xffffffcc
   a2cf8:	e59f00b4 	ldr	r0, [pc, #180]	; a2db4 <main.main+0xcc>
   a2cfc:	e58d0004 	str	r0, [sp, #4]
   a2d00:	e3a00004 	mov	r0, #4
   a2d04:	e58d0008 	str	r0, [sp, #8]
   a2d08:	e58d000c 	str	r0, [sp, #12]
   a2d0c:	ebfef5e0 	bl	60494 <runtime.makeslice>
   a2d10:	e59d0010 	ldr	r0, [sp, #16]
   a2d14:	00000000 	andeq	r0, r0, r0
   a2d18:	e3e01000 	mvn	r1, #0
   a2d1c:	e5c01000 	strb	r1, [r0]
   a2d20:	e3a01000 	mov	r1, #0
   a2d24:	e5c01001 	strb	r1, [r0, #1]
   a2d28:	e5c01002 	strb	r1, [r0, #2]
   a2d2c:	e5c01003 	strb	r1, [r0, #3]
   a2d30:	e3a01000 	mov	r1, #0
   a2d34:	e58d102c 	str	r1, [sp, #44]	; 0x2c
   a2d38:	e3a01000 	mov	r1, #0
   a2d3c:	e58d1030 	str	r1, [sp, #48]	; 0x30
   a2d40:	e58d0004 	str	r0, [sp, #4]
   a2d44:	e3a00004 	mov	r0, #4
   a2d48:	e58d0008 	str	r0, [sp, #8]

What did you see instead?

On aarch64/arm64 architecture, something strange happens. It loads x3 with -1 , but then the strb/sturh/strb instructions produce a situation where there is a 1-2-1 byte store sequence. See line 8e880 below. This does correctly set the value, but it does so in a way that likely causes an unaligned data store. The system I'm working with generates a SIGBUS for the unaligned multi-byte store. Thus I cannot use the encoding/binary PutUint32 function for writing directly to mmap'd sections, at least not without writing to a temporary first and then using copy to go from temporary to mmap'd memory.

4 single byte strb instructions would've worked fine, or a 4-byte str. But the construct below is problematic due to the unaligned access.

% objdump -d putuint32_bug_arm64 | grep -A25 '<main\.main>:'
000000000008e850 <main.main>:
   8e850:	f9400b90 	ldr	x16, [x28, #16]
   8e854:	eb3063ff 	cmp	sp, x16
   8e858:	54000489 	b.ls	8e8e8 <main.main+0x98>  // b.plast
   8e85c:	f81a0ffe 	str	x30, [sp, #-96]!
   8e860:	f81f83fd 	stur	x29, [sp, #-8]
   8e864:	d10023fd 	sub	x29, sp, #0x8
   8e868:	b0000040 	adrp	x0, 97000 <type:*+0x7000>
   8e86c:	91148000 	add	x0, x0, #0x520
   8e870:	b27e03e1 	orr	x1, xzr, #0x4
   8e874:	aa0103e2 	mov	x2, x1
   8e878:	97ff1982 	bl	54e80 <runtime.makeslice>
   8e87c:	d503201f 	nop
   8e880:	92800003 	mov	x3, #0xffffffffffffffff    	// #-1
   8e884:	39000003 	strb	w3, [x0]
   8e888:	7800101f 	sturh	wzr, [x0, #1]
   8e88c:	39000c1f 	strb	wzr, [x0, #3]
   8e890:	a904ffff 	stp	xzr, xzr, [sp, #72]
   8e894:	b27e03e1 	orr	x1, xzr, #0x4
   8e898:	aa0103e2 	mov	x2, x1
   8e89c:	97fe29bd 	bl	18f90 <runtime.convTslice>
   8e8a0:	90000043 	adrp	x3, 96000 <type:*+0x6000>
   8e8a4:	911b8063 	add	x3, x3, #0x6e0
   8e8a8:	f90027e3 	str	x3, [sp, #72]
   8e8ac:	f9002be0 	str	x0, [sp, #80]
   8e8b0:	b00005db 	adrp	x27, 147000 <runtime.itabTableInit+0x1a0>

This could be an unintended optimization, since it is generally correct code, just a little awkward for something that should "PutUint32" and problematic/illegal for certain systems.

I would research and suggest a fix myself, but I am unfamiliar with how to find the assembly code generator for low level issues like this.

[ianlancetaylor]

CC @randall77

[randall77]

The system I'm working with generates a SIGBUS for the unaligned multi-byte store.

Unaligned accesses are definitely allowed on arm64. Is this a memory-mapped device or something?

I don't see how we support enforcing aligned accesses without turning all read and write combining off. For instance,

func f(p *[4]byte) {
    p[1] = 1
    p[2] = 2
}

We want to implement that with a 16-bit write of the constant 0x201. But that write might very well be unaligned. (The general problem has nothing to do with encoding/binary specifically, although that is a package in which this situation is likely to appear.)

This definitely requires some low-level support. I don't think there's any way for us to provide that by default. Using sync/atomic or assembly seem like the two obvious workarounds.

[erifan]

I can confirm that arm64 allows unaligned access, the ARMv8-A architecture allows many types of load and store accesses to be arbitrarily aligned. Most unaligned accesses have no performance penalties. I'm also curious what kind of device this is.

[randall77]

Note that the 1-2-1 store pattern is somewhat inefficient and I think should be fixed by https://go-review.googlesource.com/c/go/+/478475

[erifan]

I am wondering if we can intrinsify this function into a 32bit store. However, the write address may still be unaligned, but we can check it before calling this function if alignment is needed.

[renthraysk]

Alignment checking is controlled by a bit in the System Control Register.

https://developer.arm.com/documentation/ddi0406/b/Application-Level-Architecture/Application-Level-Memory-Model/Alignment-support/Unaligned-data-access?lang=en

[mattypiper]

Apologies for confusion, absolutely arm64 supports unaligned access. The SIGBUS issue I experience is with a memory mapped device which does not support the unaligned access. I tried to carefully word the issue by saying things like "problematic" for "certain systems". And I understand how coalescing multiple byte writes into a single multi-byte instruction makes sense for performance. It just seemed strange for this particular go code to generate that 1-2-1 storage pattern. I would think that passing uint32(byte(0xff)) is enough to tell the compiler to generate a 4-byte store of 0x000000ff. Anyway, thanks for the quick response and consideration! I'll see if I can test the very recent patch from https://go-review.googlesource.com/c/go/+/478475.

[mattypiper]

@randall77 I can confirm that the 1-2-1 pattern is fixed in the master branch. Thanks for the link, good learning opportunity for me. There is now a single 32-bit store: str w3, [x0] in my example code. (Full assembly listing is pasted below).

And before I close the topic, I figured I'd try the 64-bit version as well. The code binary.LittleEndian.PutUint64(b, uint64(x)) produces a comparable and efficient orr x3, xzr, #0xff; str x3, [x0].

% objdump -d putuint32_bug_arm64 | grep -A25 '<main\.main>:'
000000000008b3c0 <main.main>:
   8b3c0:	f9400b90 	ldr	x16, [x28, #16]
   8b3c4:	eb3063ff 	cmp	sp, x16
   8b3c8:	54000449 	b.ls	8b450 <main.main+0x90>  // b.plast
   8b3cc:	f81a0ffe 	str	x30, [sp, #-96]!
   8b3d0:	f81f83fd 	stur	x29, [sp, #-8]
   8b3d4:	d10023fd 	sub	x29, sp, #0x8
   8b3d8:	90000060 	adrp	x0, 97000 <type:*+0x7000>
   8b3dc:	91058000 	add	x0, x0, #0x160
   8b3e0:	b27e03e1 	orr	x1, xzr, #0x4
   8b3e4:	aa0103e2 	mov	x2, x1
   8b3e8:	97ff2916 	bl	55840 <runtime.makeslice>
   8b3ec:	d503201f 	nop
   8b3f0:	b2401fe3 	orr	x3, xzr, #0xff
   8b3f4:	b9000003 	str	w3, [x0]
   8b3f8:	a904ffff 	stp	xzr, xzr, [sp, #72]
   8b3fc:	b27e03e1 	orr	x1, xzr, #0x4
   8b400:	aa0103e2 	mov	x2, x1
   8b404:	97fe344b 	bl	18530 <runtime.convTslice>
   8b408:	f0000043 	adrp	x3, 96000 <type:*+0x6000>
   8b40c:	910b8063 	add	x3, x3, #0x2e0
   8b410:	f90027e3 	str	x3, [sp, #72]
   8b414:	f9002be0 	str	x0, [sp, #80]
   8b418:	d000057b 	adrp	x27, 139000 <runtime.itabTableInit+0xc00>