Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: Crash in GC on Debian riscv64 (qemu) #54104

Open
phpeterson-usf opened this issue Jul 28, 2022 · 8 comments
Open

runtime: Crash in GC on Debian riscv64 (qemu) #54104

phpeterson-usf opened this issue Jul 28, 2022 · 8 comments
Assignees
Labels
arch-riscv compiler/runtime NeedsInvestigation
Milestone

Comments

@phpeterson-usf
Copy link

phpeterson-usf commented Jul 28, 2022

What version of Go are you using (go version)?

$ go version
go version go1.18.4 darwin/arm64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

I am cross-compiling the micro editor on my Mac targeting a Debian riscv64 guest under QEMU. The Debian guest is Linux debian 5.18.0-2-riscv64 https://github.com/zyedidia/micro/issues/1 SMP Debian 5.18.5-1 (2022-06-16) riscv64 GNU/Linux

go env Output
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/phil/Library/Caches/go-build"
GOENV="/Users/phil/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/phil/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/phil/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.18.4"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/phil/riscv64-ubuntu/micro/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/bl/ncrkxg1d6r130_5pb8mn3y3m0000gn/T/go-build3611809763=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

  1. After cloning the repo, the build command is GOOS=linux GOARCH=riscv64 CGO_ENABLED=0 make build.
  2. The resulting binary is called micro and I scp that to the Debian guest

What did you expect to see?

I expect to run the micro editor

What did you see instead?

The micro editor crashes on launch with the output below, apparently showing a crash in the Go GC. I opened an issue in the micro repo, but the author suggests that the root cause is inside Go.

Interestingly, the same micro binary runs on an Ubuntu guest Linux ubuntu 5.13.0-1026-generic #29~20.04.1-Ubuntu SMP Fri Jun 3 11:55:52 UTC 2022 riscv64 riscv64 riscv64 GNU/Linux but crashes on launch on the Debian guest.

stack trace stack trace

runtime: lfstack.push invalid packing: node=0xffffff8cc2b980 cnt=0x1 packed=0xffff8cc2b9800001 -> node=0xffff8cc2b980
fatal error: lfstack.push

runtime stack:
runtime.throw({0x5966ba, 0xc})
runtime/panic.go:992 +0x58
runtime.(*lfstack).push(0xaf62e8, 0xffffff8cc2b980)
runtime/lfstack.go:30 +0x144
runtime.(*spanSetBlockAlloc).free(...)
runtime/mspanset.go:292
runtime.(*spanSet).reset(0xaef898)
runtime/mspanset.go:265 +0xac
runtime.finishsweep_m()
runtime/mgcsweep.go:260 +0xc8
runtime.gcStart.func1()
runtime/mgc.go:664 +0x1c
runtime.systemstack()
runtime/asm_riscv64.s:133 +0x50

goroutine 1 [running]:
runtime.systemstack_switch()
runtime/asm_riscv64.s:96 +0x8 fp=0xc0004b3720 sp=0xc0004b3718 pc=0x74098
runtime.gcStart({0x0, 0x0, 0x0})
runtime/mgc.go:663 +0x500 fp=0xc0004b3798 sp=0xc0004b3720 pc=0x29188
runtime.mallocgc(0x30, 0x5576e0, 0x1)
runtime/malloc.go:1205 +0x71c fp=0xc0004b3810 sp=0xc0004b3798 pc=0x1bdb4
runtime.growslice(0x5576e0, {0x0, 0x0, 0x0}, 0x1)
runtime/slice.go:278 +0x4e0 fp=0xc0004b3860 sp=0xc0004b3810 pc=0x5e4b8
regexp/syntax.(*compiler).inst(...)
regexp/syntax/compile.go:164
regexp/syntax.(*compiler).init(...)
regexp/syntax/compile.go:83
regexp/syntax.Compile(0xc000131810)
regexp/syntax/compile.go:73 +0x9c fp=0xc0004b3920 sp=0xc0004b3860 pc=0x151214
regexp.compile({0x59c8fd, 0x14}, 0xd4, 0x0)
regexp/regexp.go:180 +0xa4 fp=0xc0004b39b8 sp=0xc0004b3920 pc=0x167c94
regexp.Compile(...)
regexp/regexp.go:135
regexp.MustCompile({0x59c8fd, 0x14})
regexp/regexp.go:315 +0x3c fp=0xc0004b3a38 sp=0xc0004b39b8 pc=0x1688ac
main.LoadInput({0xc00001c310, 0x0, 0x0})
github.com/zyedidia/micro/v2/cmd/micro/micro.go:166 +0x14c fp=0xc0004b3bb8 sp=0xc0004b3a38 pc=0x4af5d4
main.main()
github.com/zyedidia/micro/v2/cmd/micro/micro.go:319 +0x748 fp=0xc0004b3f80 sp=0xc0004b3bb8 pc=0x4b0988
runtime.main()
runtime/proc.go:250 +0x228 fp=0xc0004b3fd8 sp=0xc0004b3f80 pc=0x48480
runtime.goexit()
runtime/asm_riscv64.s:497 +0x4 fp=0xc0004b3fd8 sp=0xc0004b3fd8 pc=0x75ec4

goroutine 6 [sleep]:
time.Sleep(0x1dcd65000)
runtime/time.go:194 +0x150
github.com/zyedidia/micro/v2/internal/buffer.backupThread()
github.com/zyedidia/micro/v2/internal/buffer/backup.go:37 +0x28
created by github.com/zyedidia/micro/v2/internal/buffer.init.0
github.com/zyedidia/micro/v2/internal/buffer/backup.go:52 +0x68

goroutine 8 [syscall]:
os/signal.signal_recv()
runtime/sigqueue.go:151 +0x38
os/signal.loop()
os/signal/signal_unix.go:23 +0x1c
created by os/signal.Notify.func1.1
os/signal/signal.go:151 +0x2c

goroutine 9 [select]:
github.com/zyedidia/tcell/v2.(*tScreen).mainLoop(0xc000190000)
github.com/zyedidia/tcell/v2@v2.0.9/tscreen.go:1580 +0xe0
created by github.com/zyedidia/tcell/v2.(*tScreen).Init
github.com/zyedidia/tcell/v2@v2.0.9/tscreen.go:210 +0x6cc

goroutine 10 [IO wait]:
internal/poll.runtime_pollWait(0xffffff8cc35198, 0x72)
runtime/netpoll.go:302 +0xe0
internal/poll.(*pollDesc).wait(0xc000058e58, 0x72, 0x1)
internal/poll/fd_poll_runtime.go:83 +0x34
internal/poll.(*pollDesc).waitRead(...)
internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000058e40, {0xc0003fb000, 0x1000, 0x1000})
internal/poll/fd_unix.go:167 +0x1f0
os.(*File).read(...)
os/file_posix.go:31
os.(*File).Read(0xc00000c780, {0xc0003fb000, 0x1000, 0x1000})
os/file.go:119 +0x70
github.com/zyedidia/tcell/v2.(*tScreen).inputLoop(0xc000190000)
github.com/zyedidia/tcell/v2@v2.0.9/tscreen.go:1633 +0x94
created by github.com/zyedidia/tcell/v2.(*tScreen).Init
github.com/zyedidia/tcell/v2@v2.0.9/tscreen.go:211 +0x724

@mengzhuo
Copy link
Contributor

mengzhuo commented Jul 28, 2022

It will be fixed after this CL been merged.
https://go-review.googlesource.com/c/go/+/409055

@mengzhuo mengzhuo changed the title affected/package: Crash in GC on Debian riscv64 runtime: Crash in GC on Debian riscv64 (qemu) Jul 28, 2022
@gopherbot gopherbot added the compiler/runtime label Jul 28, 2022
@mengzhuo mengzhuo added arch-riscv compiler/runtime and removed compiler/runtime labels Jul 28, 2022
@phpeterson-usf
Copy link
Author

phpeterson-usf commented Jul 28, 2022

Thanks. I would be thrilled if this could make the 1.19 release train to support my teaching objectives.

@cherrymui cherrymui added the NeedsInvestigation label Jul 28, 2022
@cherrymui cherrymui added this to the Backlog milestone Jul 28, 2022
@cherrymui
Copy link
Member

cherrymui commented Jul 28, 2022

Can this reproduce on a real hardware?

For QEMU, I guess it would be possible to configure it to not enable SV57 mode?

@phpeterson-usf
Copy link
Author

phpeterson-usf commented Jul 28, 2022

I believe SV57 is enabled in the 5.18 kernel which is distributed with Debian/riscv64. I have looked unsuccessfully for runtime configuration for that feature. If there's a way to change that in QEMU, I don't know what it is.

I don't have an Unmatched board, and SiFive discontinued them in favor of a pre-release collaboration with Intel.

If it's too late for 1.19, the two possible workarounds I see are:

  1. Downgrade the kernel to pre-SV57
  2. Build Go from source, patching in Ian's change myself

@phpeterson-usf
Copy link
Author

phpeterson-usf commented Jul 28, 2022

FWIW, I built from go1.19rc2 + Ian's change (e9f157e.diff) and cross-compiled micro to avoid bootstrapping Go on riscv64. The resulting micro binary seems to work fine on my Debian/riscv64 guest.

@mengzhuo
Copy link
Contributor

mengzhuo commented Jul 29, 2022

Can this reproduce on a real hardware?

For QEMU, I guess it would be possible to configure it to not enable SV57 mode?

Most of existed real hardwares (that I can get) using sv39

  • Sifive unmatched(Go build farm): sv39
  • Starfive v1: sv39
  • Allwinner D1: sv39

Qemu configured with -cpu=sifive-u54 to disable SV57.

@yuzibo
Copy link

yuzibo commented Jul 29, 2022

Hi,
I just have Unmatched boards by hand and I tested it easily:

vimer@unmatched:~/go/proj/bin$ uname -a
Linux unmatched 5.18.0-2-riscv64 #1 SMP Debian 5.18.5-1 (2022-06-16) riscv64 GNU/Linux
vimer@unmatched:~/go/proj/bin$ go env
GO111MODULE=""
GOARCH="riscv64"
GOBIN=""
GOCACHE="/home/vimer/.cache/go-build"
GOENV="/home/vimer/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="riscv64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/vimer/go/proj/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/vimer/go/proj"
GOPRIVATE=""
GOPROXY="https://proxy.golang.com.cn"
GOROOT="/home/vimer/go/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/vimer/go/go/pkg/tool/linux_riscv64"
GOVCS=""
GOVERSION="go1.18.4"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2096220389=/tmp/go-build -gno-record-gcc-switches"

vimer@unmatched:~/go/proj/bin$ ./micro --version
Version: 2.0.11-dev.100
Commit hash: 225927b9
Compiled on July 29, 2022

vimer@unmatched:~/go/proj/bin$ ip a | ./micro

The micro editor runs fine in my view also.
Please let me know if there are any issues.

@gopherbot
Copy link

gopherbot commented Jul 29, 2022

Change https://go.dev/cl/409055 mentions this issue: runtime: support riscv64 SV57 mode

@mknyszek mknyszek self-assigned this Aug 3, 2022
gopherbot pushed a commit that referenced this issue Aug 4, 2022
riscv64 has SV57 mode when user-space VA is 56 bits.
Linux kernel recently got support for this mode and Go binaries started crashing as:

runtime: lfstack.push invalid packing: node=0xffffff5908a940 cnt=0x1
packed=0xffff5908a9400001 -> node=0xffff5908a940

Adjust lfstack code to use only 8 top bits of pointers on riscv64.

For context see:
https://groups.google.com/g/syzkaller-bugs/c/lU0GQTZoNQQ/m/O_c3vmE3AAAJ

Update #54104

Change-Id: Ib5d3d6a79c0c6eddf11618d73fcc8bc1832a9c25
Reviewed-on: https://go-review.googlesource.com/c/go/+/409055
Reviewed-by: Joel Sing <joel@sing.id.au>
Reviewed-by: Meng Zhuo <mzh@golangcn.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
jproberts pushed a commit to jproberts/go that referenced this issue Aug 10, 2022
riscv64 has SV57 mode when user-space VA is 56 bits.
Linux kernel recently got support for this mode and Go binaries started crashing as:

runtime: lfstack.push invalid packing: node=0xffffff5908a940 cnt=0x1
packed=0xffff5908a9400001 -> node=0xffff5908a940

Adjust lfstack code to use only 8 top bits of pointers on riscv64.

For context see:
https://groups.google.com/g/syzkaller-bugs/c/lU0GQTZoNQQ/m/O_c3vmE3AAAJ

Update golang#54104

Change-Id: Ib5d3d6a79c0c6eddf11618d73fcc8bc1832a9c25
Reviewed-on: https://go-review.googlesource.com/c/go/+/409055
Reviewed-by: Joel Sing <joel@sing.id.au>
Reviewed-by: Meng Zhuo <mzh@golangcn.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch-riscv compiler/runtime NeedsInvestigation
Projects
Status: All-But-Submitted
Development

No branches or pull requests

6 participants