runtime: go1.26 SIGILL on riscv64 with vector support

[lotheac]

Go version

go version go1.26.1 linux/riscv64

Output of go env in your module/workspace:

AR='ar'
CC='cc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='c++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='riscv64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2250152223=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='riscv64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GORISCV64='rva20u64'
GOROOT='/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/lib/go/pkg/tool/linux_riscv64'
GOVCS=''
GOVERSION='go1.26.1'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

Since go 1.26, we are seeing SIGILLs on Alpine Linux riscv64 CI on a machine that supports vector instructions (BananaPi F3 with Spacemit X60).

I have not bisected, but these commits seem like the likely causes:
75ea2d0
3406a61
d83b16f

This CPU does support vector instructions:

$ lscpu
Architecture:           riscv64
  Byte Order:           Little Endian
CPU(s):                 8
  On-line CPU(s) list:  0-7
Vendor ID:              0x710
  Model name:           Spacemit(R) X60
    CPU family:         0x8000000058000001
    Model:              0x1000000049772200
    Thread(s) per core: 1
    Core(s) per socket: 8
    Socket(s):          1
    CPU(s) scaling MHz: 100%
    CPU max MHz:        1600.0000
    CPU min MHz:        614.4000
Caches (sum of all):
  L1d:                  256 KiB (8 instances)
  L1i:                  256 KiB (8 instances)
  L2:                   1 MiB (2 instances)
$ grep isa /proc/cpuinfo |uniq
isa		: rv64imafdcv_zicbom_zicboz_zicntr_zicond_zicsr_zifencei_zihintpause_zihpm_zfh_zfhmin_zca_zcd_zba_zbb_zbc_zbs_zkt_zve32f_zve32x_zve64d_zve64f_zve64x_zvfh_zvfhmin_zvkt_sscofpmf_sstc_svinval_svnapot_svpbmt

Those vector instructions work fine when compiled into a C program, but go crashes.

I can semi-reliably trigger a SIGILL on this machine on go1.26 by compiling the following program:

~/trivial # cat main.go
package main

import "fmt"

func main() {
	fmt.Printf("hello world\n");
	return
}

It's possible (although a bit less reliable) to trigger a SIGILL even when compiling a more trivial program (func main(){return}).

What did you see happen?

~/trivial # go build -o main main.go
~/trivial # go build -o main main.go
SIGILL: illegal instruction
PC=0x14c04 m=0 sigcode=1
instruction bytes: 0x7 0x4 0x5 0x2 0x57 0x40 0x86 0x62 0x57 0xa3 0x8 0x42 0x63 0x59 0x3 0x0

goroutine 0 gp=0x142df80 m=0 mp=0x1430c80 [idle]:
indexByteBig()
	internal/bytealg/indexbyte_riscv64.s:82 +0x14 fp=0x3fd86da888 sp=0x3fd86da888 pc=0x14c04
runtime.findnull(0x963da0?)
	runtime/string.go:520 +0x54 fp=0x3fd86da8c0 sp=0x3fd86da888 pc=0x73584
runtime.gostringnocopy(...)
	runtime/string.go:544
runtime.(*moduledata).funcName(0x63278?, 0x293f2700?)
	runtime/symtab.go:762 +0x42 fp=0x3fd86da8d8 sp=0x3fd86da8c0 pc=0x7415a
runtime.funcname(...)
	runtime/symtab.go:1150
runtime.isSystemGoroutine(0x7293f2700, 0x0)
	runtime/traceback.go:1437 +0xce fp=0x3fd86da8f8 sp=0x3fd86da8d8 pc=0x7c58e
runtime.newproc1(0xa54fb0, 0x7293f21c0, 0x5ab0c, 0x0, 0x0)
	runtime/proc.go:5358 +0x162 fp=0x3fd86da960 sp=0x3fd86da8f8 pc=0x634ba
runtime.newproc.func1()
	runtime/proc.go:5299 +0x2e fp=0x3fd86da988 sp=0x3fd86da960 pc=0x63326
runtime.systemstack(0x1430c80)
	runtime/asm_riscv64.s:230 +0x4a fp=0x3fd86da990 sp=0x3fd86da988 pc=0x8bcca

goroutine 1 gp=0x7293f21c0 m=0 mp=0x1430c80 [running, locked to thread]:
runtime.systemstack_switch()
	runtime/asm_riscv64.s:193 +0x8 fp=0x729446608 sp=0x729446600 pc=0x8bc70
runtime.newproc(0x6812e?)
	runtime/proc.go:5298 +0x3a fp=0x729446638 sp=0x729446608 pc=0x632ea
runtime.init.6()
	runtime/proc.go:363 +0x24 fp=0x729446648 sp=0x729446638 pc=0x5ab0c
runtime.doInit1(0x12bf9a0)
	runtime/proc.go:8103 +0xa6 fp=0x729446760 sp=0x729446648 pc=0x6812e
runtime.doInit(...)
	runtime/proc.go:8070
runtime.main()
	runtime/proc.go:203 +0xfa fp=0x7294467d8 sp=0x729446760 pc=0x5a792
runtime.goexit({})
	runtime/asm_riscv64.s:630 +0x4 fp=0x7294467d8 sp=0x7294467d8 pc=0x8d4dc

ra  0x73584	sp  0x3fd86da888
gp  0x2abfac7800	tp  0x3f9b2edd98
t0  0x100	t1  0x0
t2  0xeb7068	s0  0x45f
s1  0x56ff	a0  0xa86aa6
a1  0x55a	a2  0x0
a3  0xa86aa6	a4  0x0
a5  0x1	a6  0x0
a7  0x1	s2  0x1f0
s3  0x2	s4  0x2
s5  0x14514c0	s6  0x8
s7  0x10	s8  0x2
s9  0x10	s10 0x3fd86da8e0
s11 0x142df80	t3  0x18
t4  0xa1b907517bb8e	t5  0xffffffffffffffff
t6  0xfff	pc  0x14c04
go: error obtaining buildID for go tool compile: exit status 2

What did you expect to see?

I expected a successful compilation.

Curiously, GODEBUG=asyncpreemptoff=1 seems to work around the issue: if this is set in env for go build, the SIGILLs go away entirely; I can even recompile go itself with this set.

[gabyhelp]

Related Issues

• simd/archsimd: SIGILL: illegal instruction #77533 (closed)
• simd/archsimd: tests SIGILL on unsupported cpus, missing feature check #76866 (closed)
• runtime,cmd/compile: `SIGILL` on linux/riscv64 #61833 (closed)
• runtime: "SIGILL: illegal instruction" at arm64 v8.0 and use go routing #70987 (closed)
• cmd/go: Compiler crashes with GOTOOLCHAIN=go1.26rc1 GOEXPERIMENT=simd on Apple M1 #76879 (closed)
• cmd/compile/internal/amd64: TestGoAMD64v1 test failures (on some machines) #77674 (closed)
• strings: SIGILL on s390x on z196 #17032 (closed)
• cmd/compile: SIGILL when building golang.org/x/sys/unix on openbsd/arm64 #42517 (closed)
• cmd/compile: SIGILL when cross-compiling from netbsd/arm64 to linux/mips #52651
• cmd/compile,runtime: SIGILL on `linux-loong64-3a5000` #53093

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[randall77]

The instruction in question is, I believe

VLE8V	(X10), V8

It is right after another vector instruction, and that one succeeded.
If I understand, it is a pretty bog-standard vector load instruction (of VLEN bytes?).

We test HWPROBE_IMA_V before using the instruction, so presumably your chip reports that V is supported.

So is your chip erroneously reporting V support? Or does VLE8V require something more that just V, that your chip does not have? Or is our V test not correct somehow?

@golang/riscv64

[randall77]

I don't see how GODEBUG=asyncpreemptoff=1 could matter. Unless there is an instruction in our preempt path (runtime/preempt_arm64.s) that needs to be checked for support? But then I don't see why it would trigger only at that instruction. Does the illegal instruction only ever occur at that particular PC, or have you seen it elsewhere?

[randall77]

Could it be that the kernel is upset about using vector instructions in signal handlers? We've had that problem with other OSes (plan9, particularly). The stack trace doesn't seem to indicate that, though.

[lotheac]

The instruction in question is, I believe

VLE8V	(X10), V8

It is right after another vector instruction, and that one succeeded. If I understand, it is a pretty bog-standard vector load instruction (of VLEN bytes?).

We test HWPROBE_IMA_V before using the instruction, so presumably your chip reports that V is supported.

Correct, V is supported. I ran some test C programs with inline asm to verify those instructions do work.

So is your chip erroneously reporting V support? Or does VLE8V require something more that just V, that your chip does not have? Or is our V test not correct somehow?

I don't see how GODEBUG=asyncpreemptoff=1 could matter. Unless there is an instruction in our preempt path (runtime/preempt_arm64.s) that needs to be checked for support? But then I don't see why it would trigger only at that instruction. Does the illegal instruction only ever occur at that particular PC, or have you seen it elsewhere?

this is the only rv64 machine with vector instruction support that I have access to, but it ostensibly is the same CPU and board that 75ea2d0 et al were benchmarked on.

Could it be that the kernel is upset about using vector instructions in signal handlers? We've had that problem with other OSes (plan9, particularly). The stack trace doesn't seem to indicate that, though.

I'm by no means an expert here, in either riscv64 or go, but my best guess it that it could have something to do with register saving on goroutine context switches. I tested a C program which executes vector insns and sched_yield() in a loop on four threads. It does not crash; this tells me the kernel is likely handling context switches between vector insns correctly, but perhaps Go is not.

This is the C program I tested; generated with LLM assistance so buyer beware :)

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <unistd.h>
#include <sched.h>

#define NUM_ITERATIONS 10000000

// This function uses inline assembly to force vector instructions and context switching
void* vector_stress_thread(void* arg) {
    int thread_id = *(int*)arg;
    int dummy;

    printf("Thread %d starting stress test...\n", thread_id);

    for (int i = 0; i < NUM_ITERATIONS; i++) {
        // We use inline assembly to emit vector instructions directly.
        // This bypasses <riscv_vector.h> and tests the Kernel's support
        // for saving/restoring the vector register state (v0).

        asm volatile (
            // 1. Setup: Load a value into general purpose register a0 (x10)
            "mv a0, %1\n\t"

            // 2. Vector Configuration: vsetvli t0, a0, e32, m1
            // Sets Vector Length (VL) to the value in a0 (which is 'thread_id', sanitized),
            // Element Width to 32-bits (e32), and LMUL to 1 (m1).
            // This instruction is critical: it allocates vector resources.
            "li a0, 16\n\t"             // Set AVL to 16
            "vsetvli t0, a0, e32, m1\n\t"

            // 3. Vector Operation: vmv.v.x v0, a0
            // Move the scalar value in a0 into the first vector register v0.
            // This "dirties" the vector state, forcing the kernel to save it later.
            "mv a0, %1\n\t"             // Reload thread_id
            "vmv.v.x v0, a0\n\t"

            // 4. Vector Operation: vadd.vv v0, v0, v0
            // Add v0 to itself. This ensures the vector unit is actively processing.
            "vadd.vv v0, v0, v0\n\t"

            // 5. Cleanup (optional): Clear vector config to exit cleanly
            // "vsetvli t0, zero, e8, m1\n\t"

            : "=r"(dummy)
            : "r"(thread_id)
            : "a0", "t0", "v0", "memory"
        );

        // Force Context Switch
        // If the kernel cannot handle vector state (v0) switching for this CPU,
        // this call will likely trigger the crash.
        sched_yield();
    }

    printf("Thread %d finished.\n", thread_id);
    return NULL;
}

int main() {
    pthread_t t1, t2, t3, t4;
    int id1 = 1;
    int id2 = 2;
    int id3 = 3;
    int id4 = 4;

    printf("Starting Vector Context Switch Stress Test...\n");

    pthread_create(&t1, NULL, vector_stress_thread, &id1);
    pthread_create(&t2, NULL, vector_stress_thread, &id2);
    pthread_create(&t3, NULL, vector_stress_thread, &id3);
    pthread_create(&t4, NULL, vector_stress_thread, &id4);

    pthread_join(t1, NULL);
    pthread_join(t2, NULL);
    pthread_join(t3, NULL);
    pthread_join(t4, NULL);

    printf("SUCCESS: System survived vector context switching.\n");
    return 0;
}

it also doesn't crash without the sched_yield() call.

[mengzhuo]

I've tested on Bianbu OS (SpacemiT's own kernel) no error found.

The SpacemiT X60 is a heterogeneous SoC, where 4 out of its 8 cores cannot execute certain vector instructions. The kernel provided by SpacemiT handles this by trapping the corresponding instructions and migrating the relevant context to a suitable core. This issue might be related to Alpine not having the necessary patches applied. As for your C test program, it just so happens to use 4 threads... so it might not have encountered a similar issue.

uname -a

Linux bb-worker-1 6.6.63 #2.2.7.3 SMP PREEMPT Fri Aug 15 06:14:36 UTC 2025 riscv64 riscv64 riscv64 GNU/Li
nux

lscpu

Model name:            spacemit,x60
  CPU family:          0x8000000058000001
  Model:               0x1000000049772200
  Thread(s) per core:  1
  Core(s) per socket:  1
  Socket(s):           1
  CPU(s) scaling MHz:  100%
  CPU max MHz:         1600.0000
  CPU min MHz:         614.4000
Caches (sum of all):
  L1d:                 256 KiB (8 instances)
  L1i:                 256 KiB (8 instances)
  L2:                  1 MiB (2 instances)

grep isa /proc/cpuinfo |uniq

isa             : rv64imafdcv_zicbom_zicboz_zicntr_zicond_zicsr_zifencei_zihintpause_zihpm_zfh_zfhmin_zca
_zcd_zba_zbb_zbc_zbs_zkt_zve32f_zve32x_zve64d_zve64f_zve64x_zvfh_zvfhmin_zvkt_sscofpmf_sstc_svinval_svnap
ot_svpbmt_ime
isa             : rv64imafdcv_zicbom_zicboz_zicntr_zicond_zicsr_zifencei_zihintpause_zihpm_zfh_zfhmin_zca
_zcd_zba_zbb_zbc_zbs_zkt_zve32f_zve32x_zve64d_zve64f_zve64x_zvfh_zvfhmin_zvkt_sscofpmf_sstc_svinval_svnap
ot_svpbmt

os-release

root@bb-worker-1:~# cat /etc/os-release
PRETTY_NAME="Bianbu 3.0.1"
NAME="Bianbu"
VERSION_ID="3.0.1"
VERSION="3.0.1 (Plucky Puffin)"
VERSION_CODENAME=plucky
ID=bianbu
ID_LIKE=debian
HOME_URL="https://bianbu.spacemit.com"
SUPPORT_URL="https://bianbu.spacemit.com"
BUG_REPORT_URL="https://ticket.spacemit.com"
PRIVACY_POLICY_URL="https://www.spacemit.com/privacy-policy"
UBUNTU_CODENAME=plucky
LOGO=bianbu-logo

Go env

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='riscv64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1401299
106=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='riscv64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GORISCV64='rva20u64'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_riscv64'
GOVCS=''
GOVERSION='go1.26.1'
GOWORK=''
PKG_CONFIG='pkg-config'
root@bb-worker-1:~#

[lotheac]

I've tested on Bianbu OS (SpacemiT's own kernel) no error found.

The SpacemiT X60 is a heterogeneous SoC, where 4 out of its 8 cores cannot execute certain vector instructions. The kernel provided by SpacemiT handles this by trapping the corresponding instructions and migrating the relevant context to a suitable core. This issue might be related to Alpine not having the necessary patches applied. As for your C test program, it just so happens to use 4 threads... so it might not have encountered a similar issue.

it has the same behavior with 8 threads as well - no crash.

uname -a

Linux bb-worker-1 6.6.63 #2.2.7.3 SMP PREEMPT Fri Aug 15 06:14:36 UTC 2025 riscv64 riscv64 riscv64 GNU/Li
nux

we're also using a kernel with spacemit patches. https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/testing/linux-spacemit/APKBUILD?ref_type=heads#L17

$ uname -a
Linux alpine-bpi-f3 6.6.53-1-spacemit #2-Alpine SMP 2024-11-16 07:35:39 riscv64 GNU/Linux

@ncopa maintains this kernel port (and actually owns this machine) -- we can ask him if you have any suspicions about the kernel patches/configs being incorrect (or the hardware having issues)

lscpu

Model name:            spacemit,x60
  CPU family:          0x8000000058000001
  Model:               0x1000000049772200
  Thread(s) per core:  1
  Core(s) per socket:  1
  Socket(s):           1
  CPU(s) scaling MHz:  100%
  CPU max MHz:         1600.0000
  CPU min MHz:         614.4000
Caches (sum of all):
  L1d:                 256 KiB (8 instances)
  L1i:                 256 KiB (8 instances)
  L2:                  1 MiB (2 instances)

grep isa /proc/cpuinfo |uniq

isa             : rv64imafdcv_zicbom_zicboz_zicntr_zicond_zicsr_zifencei_zihintpause_zihpm_zfh_zfhmin_zca
_zcd_zba_zbb_zbc_zbs_zkt_zve32f_zve32x_zve64d_zve64f_zve64x_zvfh_zvfhmin_zvkt_sscofpmf_sstc_svinval_svnap
ot_svpbmt_ime
isa             : rv64imafdcv_zicbom_zicboz_zicntr_zicond_zicsr_zifencei_zihintpause_zihpm_zfh_zfhmin_zca
_zcd_zba_zbb_zbc_zbs_zkt_zve32f_zve32x_zve64d_zve64f_zve64x_zvfh_zvfhmin_zvkt_sscofpmf_sstc_svinval_svnap
ot_svpbmt

here we have some differences: it looks like the machine I am running on does NOT have heterogeneous cores according to lscpu or cpuinfo 🤔

$ lscpu
Architecture:           riscv64
  Byte Order:           Little Endian
CPU(s):                 8
  On-line CPU(s) list:  0-7
Vendor ID:              0x710
  Model name:           Spacemit(R) X60
    CPU family:         0x8000000058000001
    Model:              0x1000000049772200
    Thread(s) per core: 1
    Core(s) per socket: 8
    Socket(s):          1
    CPU(s) scaling MHz: 100%
    CPU max MHz:        1600.0000
    CPU min MHz:        614.4000
Caches (sum of all):
  L1d:                  256 KiB (8 instances)
  L1i:                  256 KiB (8 instances)
  L2:                   1 MiB (2 instances)

$ grep isa /proc/cpuinfo|uniq
isa		: rv64imafdcv_zicbom_zicboz_zicntr_zicond_zicsr_zifencei_zihintpause_zihpm_zfh_zfhmin_zca_zcd_zba_zbb_zbc_zbs_zkt_zve32f_zve32x_zve64d_zve64f_zve64x_zvfh_zvfhmin_zvkt_sscofpmf_sstc_svinval_svnapot_svpbmt

[markdryan]

RVV 1.0 only works reliably on Kernels 6.9 and upwards. There's a bug (and a patch) related to signal handling that causes the sort of crashes you're seeing, in earlier kernels. Can you check to see if your custom kernel has this patch?

Also, you could try the following reproducer.

https://github.com/robehn/hacks/blob/main/copy/test.c

Just build it, run it and press CTRL+C. Repeat several times to reproduce the error.

[lotheac]

RVV 1.0 only works reliably on Kernels 6.9 and upwards. There's a bug (and a patch) related to signal handling that causes the sort of crashes you're seeing, in earlier kernels. Can you check to see if your custom kernel has this patch?

Also, you could try the following reproducer.

https://github.com/robehn/hacks/blob/main/copy/test.c

Just build it, run it and press CTRL+C. Repeat several times to reproduce the error.

thanks, I think you're hitting the nail on the head. we don't have that patch, and your reproducer dies.

alpine-bpi-f3:/var/home/lotheac/hacks/copy$ ./test_copy
^CIllegal instruction
alpine-bpi-f3:/var/home/lotheac/hacks/copy$ ./test_copy
^CIllegal instruction
alpine-bpi-f3:/var/home/lotheac/hacks/copy$ ./test_copy
^C^CIllegal instruction

therefore I think this issue can be closed since it doesn't seem like a go issue after all.