Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: cross compile for OpenBSD arm64 fails with SIGILL Illegal instruction #66040

Closed
ikmckenz opened this issue Feb 29, 2024 · 17 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-OpenBSD
Milestone

Comments

@ikmckenz
Copy link

ikmckenz commented Feb 29, 2024

Go version

go version go1.22.0 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/ian/Library/Caches/go-build'
GOENV='/Users/ian/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/ian/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/ian/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.22.0/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.22.0/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.0'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/gh/8flwtdfn4c526_bx0m7ht8380000gn/T/go-build3231430879=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Cross compiling a simple hello world program for OpenBSD arm64 fails with SIGILL Illegal instruction.

Hello world code:

package main

import "fmt"

func main() {
    fmt.Println("hello world")
}

Compiled with GOOS=openbsd GOARCH=arm64 go build hello.go from a different machine, and run on an OpenBSD VM running on my MacBook with an M3 Max chip: OpenBSD openbsd-dev.my.domain 7.4 GENERIC.MP#2273 arm64 fails with illegal instruction.

What did you see happen?

Natively building hello world on OpenBSD arm64 works fine, and that machine does have GOOS=openbsd and GOARCH=arm64 defined in the go env, but building the above code with the above command from my darwin/arm64 machine running go1.22.0 as above, and also my linux/amd64 machine running go 1.21.7 produces the illegal instruction error.

GDB on openbsd with the build from linux:

Program received signal SIGILL, Illegal instruction.
_rt0_arm64_openbsd () at /usr/lib/go-1.21/src/runtime/rt0_openbsd_arm64.s:15
15	/usr/lib/go-1.21/src/runtime/rt0_openbsd_arm64.s: No such file or directory.

GDB on openbsd from the build from my Mac:

Program received signal SIGILL, Illegal instruction.
_rt0_arm64_openbsd ()
    at /opt/homebrew/Cellar/go/1.22.0/libexec/src/runtime/rt0_openbsd_arm64.s:15
15	/opt/homebrew/Cellar/go/1.22.0/libexec/src/runtime/rt0_openbsd_arm64.s: No such file or directory.

GDB on openbsd from mac with GOOS=openbsd GOARCH=arm64 go build -trimpath hello.go:

Program received signal SIGILL, Illegal instruction.
_rt0_arm64_openbsd () at runtime/rt0_openbsd_arm64.s:15
15	runtime/rt0_openbsd_arm64.s: No such file or directory.

What did you expect to see?

Note that building the above hello world code into an OpenBSD amd64 binary from both my darwin/arm64 machine and my linux/amd64 machine works and the resulting binary works on a separate OpenBSD amd64 machine I have without issue, so something seems to be wrong with OpenBSD arm64 specifically. I do see openbsd/arm64 in the output of go tool dist list, so I would expect it to work.

@randall77
Copy link
Contributor

Can you print the stack pointer at those failures? It might be incorrectly aligned.

https://stackoverflow.com/questions/63003217/behaviors-of-stack-pointer-in-arm64

@ikmckenz
Copy link
Author

Included a couple extra commands of gdb output in case it helps:

(gdb) r
Starting program: /home/ian/hellomactrimpath 

Program received signal SIGILL, Illegal instruction.
_rt0_arm64_openbsd () at runtime/rt0_openbsd_arm64.s:15
15	runtime/rt0_openbsd_arm64.s: No such file or directory.
(gdb) bt
#0  _rt0_arm64_openbsd () at runtime/rt0_openbsd_arm64.s:15
(gdb) info frame
Stack level 0, frame at 0x68547a1e80:
 pc = 0x73960 in _rt0_arm64_openbsd (runtime/rt0_openbsd_arm64.s:15); 
    saved pc = 0x0
 source language asm.
 Arglist at 0x68547a1e80, args: 
 Locals at 0x68547a1e80, Previous frame's sp is 0x68547a1e80
(gdb) p $sp
$1 = (void *) 0x68547a1e80
(gdb) x/40x $sp
0x68547a1e80:	0x00000001	0x00000000	0x547a1fa0	0x00000068
0x68547a1e90:	0x00000000	0x00000000	0x547a1fbb	0x00000068
0x68547a1ea0:	0x547a1fd8	0x00000068	0x547a1fe4	0x00000068
0x68547a1eb0:	0x547a1ff2	0x00000068	0x547a2001	0x00000068
0x68547a1ec0:	0x547a2014	0x00000068	0x547a201f	0x00000068
0x68547a1ed0:	0x547a2032	0x00000068	0x547a2053	0x00000068
0x68547a1ee0:	0x547a20b2	0x00000068	0x547a20c1	0x00000068
0x68547a1ef0:	0x547a20d5	0x00000068	0x547a20de	0x00000068
0x68547a1f00:	0x547a2110	0x00000068	0x00000000	0x00000000
0x68547a1f10:	0x00000003	0x00000000	0x00010040	0x00000000

@jrick
Copy link
Contributor

jrick commented Feb 29, 2024

this is possibly BTI as it appears that you are running -current. I can't reproduce it on an arm64 machine without BTI (and unfortunately don't have one with it).

@jrick
Copy link
Contributor

jrick commented Feb 29, 2024

Go from ports is patched to mark executables NOBTCFI; something similar is going to be needed in upstream Go as well.

openbsd/ports@71f21f5

@kettenis
Copy link

Upstream should really implement support for PAC and/or BTI.

@ikmckenz
Copy link
Author

This is a fresh install of 7.4 from install74.img, not -current. Running on a VM on MacOS via UTM (a QEMU/HVF wrapper) with native virtualization, not emulation:

# uname -a
OpenBSD openbsd-dev.my.domain 7.4 GENERIC.MP#2273 arm64

Although separately yes, upstream support for BTI and PAC would be great :)

Is there a way I could use the go build flags to test if this is BTI related?

@jrick
Copy link
Contributor

jrick commented Feb 29, 2024

The ports patch is relevant to the system's native linker (ld.lld), not cross compiling. It is adding a PT_OPENBSD_NOBTCFI section but I'm not sure if this can be done without modifications to cmd/link.

If you update to -current and use ports Go, you could probably test this hypothesis comparing go build -ldflags=-linkmode=external to go build -ldflags=-linkmode=internal. The former would include the PT_OPENBSD_NOBTCFI section and the latter would not.

@sthen
Copy link

sthen commented Feb 29, 2024

BTI is used on amd64/arm64 if supported by the cpu in 7.4, no need for -current.

@sthen
Copy link

sthen commented Feb 29, 2024

The ports patch is relevant to the system's native linker (ld.lld), not cross compiling. It is adding a PT_OPENBSD_NOBTCFI section but I'm not sure if this can be done without modifications to cmd/link.

If you update to -current and use ports Go, you could probably test this hypothesis comparing go build -ldflags=-linkmode=external to go build -ldflags=-linkmode=internal. The former would include the PT_OPENBSD_NOBTCFI section and the latter would not.

IIRC we have patched for both internal and external linkers.

@jrick
Copy link
Contributor

jrick commented Feb 29, 2024

Ah, yes that true: https://github.com/openbsd/ports/blob/master/lang/go/patches/patch-src_cmd_link_internal_ld_elf_go

So that experiment won't work, both will have the section.

In any case, this seems to be the issue.

@ikmckenz
Copy link
Author

ikmckenz commented Mar 1, 2024

I think the issue is my CPU is too new. Reproducing the exact same test on an M1 instead of an M3 Max works without issue, and I now see that only M1 and M2 generation chips are currently supported on the OpenBSD hardware support page.

Curiously, it doesn't seem to be BTI. My M3 Max doesn't show BT in the CPU info in dmesg with the UTM virtualization:

cpu0 at mainbus0 mpidr 0: Unknown, MIDR 0x610f0000
cpu0: 192KB 64b/line 6-way L1 PIPT I-cache, 128KB 64b/line 8-way L1 D-cache
cpu0: 16384KB 128b/line 16-way L2 cache
cpu0: TLBIOS+IRANGE,TS+AXFLAG,FHM,DP,SHA3,RDM,Atomic,CRC32,SHA2+SHA512,SHA1,AES+PMULL,SPECRES,SB,FRINTTS,GPI,LRCPC+LDAPUR,FCMA,JSCVT,API+PAC,DPB,SpecSEI,PAN+ATS1E1+EPAN,LO,HPDS,CSV3,CSV2,DIT

And on the M1 it also doesn't show up:

cpu0 at mainbus0 mpidr 0: Unknown, MIDR 0x610f0000
cpu0: 192KB 64b/line 6-way L1 PIPT I-cache, 128KB 64b/line 8-way L1 D-cache
cpu0: 12288KB 128b/line 12-way L2 cache
cpu0: TLBIOS+IRANGE,TS+AXFLAG,FHM,DP,SHA3,RDM,Atomic,CRC32,SHA2+SHA512,SHA1,AES+PMULL,SPECRES,SB,FRINTTS,GPI,LRCPC+LDAPUR,FCMA,JSCVT,API+PAC,DPB,SpecSEI,PAN+ATS1E1,LO,HPDS,CSV3,CSV2,DIT

Sorry about the false alarm all, I'll close the issue now and wait patiently for M3 support in OpenBSD :)

@ikmckenz ikmckenz closed this as completed Mar 1, 2024
@jrick
Copy link
Contributor

jrick commented Mar 1, 2024

BTI support was introduced on the M2. It's not unexpected that you wouldn't see this crash on M1.

I don't know why BT is missing from the M3's dmesg. OpenBSD (nor Asahi) does not support booting the platform natively yet.

Perhaps enforcement is still occurring despite it being missing from dmesg.

I would reopen the issue. In the short term, NOBTCFI will work around this, but OpenBSD would prefer that BTI (and IBT on amd64) be properly implemented, and for these instructions to not only be emitted for OpenBSD targets, but all platforms.

@ikmckenz ikmckenz reopened this Mar 1, 2024
@jrick
Copy link
Contributor

jrick commented Mar 1, 2024

NOBTCFI will work around this

Can you build the hello world program from ports Go to confirm this? It should run and you should also see

$ readelf -l ./hello | grep NOBTCF
  OPENBSD_NOBTCF 0x0000000000000000 0x0000000000000000 0x0000000000000000

@ikmckenz
Copy link
Author

ikmckenz commented Mar 1, 2024

Confirmed when building with go from ports it runs fine, and that is the exact output from readelf.

@ikmckenz
Copy link
Author

ikmckenz commented Mar 1, 2024

Also confirmed the issue is BTI being enforced: I upgraded the VM to the most recent snapshot which includes the recently added code meant to identify this exact scenario, and there it is at the end of the output of kdump clear as day:

 75545 hello    PSIG  SIGILL SIG_DFL code=ILL_BTCFI addr=0x73960 trapno=905969665
 75545 hello    NAMI  "hello.core"

On 7.4 the code was just ILL_ILLOPC.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/568435 mentions this issue: cmd/link,debug/elf: mark Go binaries with no branch target CFI on openbsd

@seankhliao seankhliao changed the title Hello World cross build for OpenBSD arm64 fails with SIGILL Illegal instruction cmd/compile: cross compile for OpenBSD arm64 fails with SIGILL Illegal instruction Mar 1, 2024
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Mar 1, 2024
@seankhliao seankhliao added OS-OpenBSD NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Mar 1, 2024
@mknyszek mknyszek added this to the Backlog milestone Mar 6, 2024
@ikmckenz
Copy link
Author

ikmckenz commented Mar 13, 2024

Just chiming in to confirm that 1e43391 has fixed the issue for me. Compiling go from source after the commit on Linux, and cross building hello world for openbsd/arm64, the resulting binary runs correctly and readelf shows the OPENBSD_NOBTCF section. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-OpenBSD
Projects
Development

No branches or pull requests

8 participants