hash/crc32: panic on arm64 with go1.21.0 when indexing slice

[Slessi]

What version of Go are you using (go version)?

$ go version
go version go1.21.0 darwin/arm64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/edward/Library/Caches/go-build'
GOENV='/Users/edward/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/edward/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/edward/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.21.0/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.21.0/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.21.0'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/ld/l69fjx9x3gl6qfx056xt89hr0000gp/T/go-build4293017448=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

This code panics on go 1.21 on arm64 but runs fine on other platforms or go 1.20 arm64:

It doesn't occur if I add a fmt.Println(index) or debug the application, before calling colors[index]

package main

import (
	"fmt"
	"hash/crc32"
)

func pickColor(userID string) string {
	colors := []string{
		"ffd179",
		"f388a8",
		"b786d2",
		"bed3f4",
		"88a0f3",
		"6fe8d9",
		"e86f85",
	}
	hash := crc32.ChecksumIEEE([]byte(userID))
	index := hash % uint32(len(colors))
	return colors[index]
}

func main() {
	fmt.Println(pickColor("jHjDiYOWTOS53K9t3XbOU7QXAwD3"))
}

What did you expect to see?

A string printed

What did you see instead?

panic: runtime error: index out of range [-4294967295]

goroutine 1 [running]:
main.pickColor({0x10494b15b?, 0x0?})
        /Users/edward/Desktop/go_bug.go:20 +0x118
main.main()
        /Users/edward/Desktop/go_bug.go:24 +0x28
exit status 2

[ubombi]

Smaller example

package main

import "hash/crc32"

func main() {
	colors := []string{"ffd179", "b786d2", "bed3f4"}

	// has is uint32(754903878), but panic occures only during crc32 call
	// and not, when you set this value explicitely
	hash := crc32.ChecksumIEEE([]byte("jHjDiYOWTOS53K9t3XbOU7QXAwD3"))
	index := hash % uint32(3)

	_ = colors[index] // PANIC: index out of range -4294967296
	return
}

The magic uint64 number -4294967296 is 0xFFFFFFFF 00000000, but all variables are 32bit unsigned ints. Thus
dropping high bits, shall be 0 or 0x00000000, however, due to optimizations, go compares -4294967296 with array length and panics.

More context

during execution, around index := hash % uint32(len(colors)) :

<main.main+108> sub x0, x0, x1

x0: 0xFFFFFFFF 2CFEEB46 -- register, which would be used as array index is .
x1: 0x2CFEEB46 -- (754903878) value needed for reminder calculation.

After substracting x1 from x0 part, we end with 0xFFFFFFFF00000000, where previous XORing set high bit's to 1.
As result out of bounds check fails, comparing remainder with array length fails and jumps to Panic.

uint32 type is ignored here. Everywhere.

Panics on go1.21.0 linux/arm64
Works on go1.21.0 linux/amd64

[ubombi]

According to docs & playground proof:

^x bitwise complement is m ^ x with m = "all bits set to 1" for unsigned x and m = -1 for signed x

Following call makes sets x0 to -1, effectively 0xfffffffffffffff

return ^ieeeUpdate(^crc, p)     // hash/crc32.archUpdateIEEE mvn x0 x0

[dmitshur]

CC @golang/security.

[ubombi]

No dependencies example

./bug.s

#include "textflag.h"

// func simple(crc uint32, p []byte) uint32
TEXT ·simple(SB),NOSPLIT,$0-36
	MOVWU	crc+0(FP), R9  // CRC value
	MOVWU	R9, ret+32(FP)
	RET

./bug.go

package main

func simple(crc uint32, p []byte) uint32

func main() {
	colors := []string{ "", "", ""}
	hash := archUpdateIEEE(0, []byte("jHjDiYOWTOS53K9t3XbOU7QXAwD3"))
	index := hash % uint32(3)
	_ = colors[index]
	return
}

func archUpdateIEEE(crc uint32, p []byte) uint32 {
	return ^simple(^crc, p)
}

[bcmills]

Since you can reproduce the problem in assembly, it seems likely that this is a compiler or runtime change.

Can you build the Go repo from source and use git bisect to find the exact commit at which the regression began?

[bcmills]

(attn @golang/compiler @golang/arm)