runtime: panic while TestASAN with "stack growth after fork" #50391

mengzhuo · 2021-12-30T02:49:38Z

What version of Go are you using (`go version`)?

$ go version
go version devel go1.18-91e782106e Wed Dec 29 04:10:07 2021 +0000 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/root/godev"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/root/godev/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="devel go1.18-91e782106e Wed Dec 29 04:10:07 2021 +0000"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2639948742=/tmp/go-build -gno-record-gcc-switches"
GOROOT/bin/go version: go version devel go1.18-91e782106e Wed Dec 29 04:10:07 2021 +0000 linux/amd64
GOROOT/bin/go tool compile -V: compile version devel go1.18-91e782106e Wed Dec 29 04:10:07 2021 +0000
uname -sr: Linux 5.10.0-9-amd64
Distributor ID:	Debian
Description:	Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye
/lib/x86_64-linux-gnu/libc.so.6: GNU C Library (Debian GLIBC 2.31-13+deb11u2) stable release version 2.31.
gdb --version: GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git

What did you do?

cd misc/cgo/testsanitizers/ && go test -asan -v

What did you expect to see?

PASSED

What did you see instead?

fatal error: stack growth after fork
runtime stack:

runtime.throw({0x5de782?, 0x200000008?})

/root/godev/src/runtime/panic.go:992 +0x71

runtime.newstack()

/root/godev/src/runtime/stack.go:964 +0xc07

runtime.morestack()

/root/godev/src/runtime/asm_amd64.s:547 +0x8b
goroutine 6 [running]:

runtime.asanwrite(0x10c0001091c0?, 0x20?)

/root/godev/src/runtime/asan.go:35 +0x5c fp=0x10c000109140 sp=0x10c000109138 pc=0x405a1c

runtime.sigaction(0x1, 0x10c0001091c0, 0x0)

/root/godev/src/runtime/cgo_sigaction.go:31 +0x32 fp=0x10c0001091a8 sp=0x10c000109140 pc=0x405db2

runtime.setsig(0x6d95c8?, 0x0?)

/root/godev/src/runtime/os_linux.go:468 +0x76 fp=0x10c0001091f0 sp=0x10c0001091a8 pc=0x4342f6

runtime.clearSignalHandlers()

/root/godev/src/runtime/signal_unix.go:261 +0x35 fp=0x10c000109218 sp=0x10c0001091f0 pc=0x44bd95

syscall.runtime_AfterForkInChild()

/root/godev/src/runtime/proc.go:4186 +0x1a fp=0x10c000109250 sp=0x10c000109218 pc=0x4641ba

syscall.forkAndExecInChild1(0x10c000018210, {0x10c00000e2a0, 0x4, 0x80000?}, {0x10c00012c3c0, 0x16, 0x8?}, 0x0, 0x0, 0x10c0001097a0, ...)

/root/godev/src/syscall/exec_linux.go:296 +0x76a fp=0x10c000109488 sp=0x10c000109250 pc=0x4748ea

syscall.forkAndExecInChild(0x10c0000535a8?, {0x10c00000e2a0?, 0x7f3ca2c35ed8?, 0x340?}, {0x10c00012c3c0?, 0x440ec5?, 0x10c00010b0e0?}, 0x300000002?, 0x10c00010b0e0?, 0x10c0001097a0, ...)

/root/godev/src/syscall/exec_linux.go:84 +0x65 fp=0x10c000109548 sp=0x10c000109488 pc=0x473fc5

syscall.forkExec({0x10c000018180, 0x14}, {0x10c00007e540, 0x3, 0x3}, 0x10c000012680?)

/root/godev/src/syscall/exec_unix.go:209 +0x51c fp=0x10c0001096c8 sp=0x10c000109548 pc=0x476abc

syscall.StartProcess(...)

/root/godev/src/syscall/exec_unix.go:255

os.startProcess({0x10c000018180, 0x14}, {0x10c00007e540, 0x3, 0x3}, 0x10c000109a90)

/root/godev/src/os/exec_posix.go:54 +0x5cd fp=0x10c0001097f8 sp=0x10c0001096c8 pc=0x49c7cd

os.StartProcess({0x10c000018180, 0x14}, {0x10c00007e540, 0x3, 0x3}, 0x10c0000181b0?)

/root/godev/src/os/exec.go:109 +0x5a fp=0x10c000109840 sp=0x10c0001097f8 pc=0x49c11a

os/exec.(*Cmd).Start(0x10c000012340)

/root/godev/src/os/exec/exec.go:422 +0xa1d fp=0x10c000109ae8 sp=0x10c000109840 pc=0x51a0fd

os/exec.(*Cmd).Run(0x518265?)

/root/godev/src/os/exec/exec.go:338 +0x1e fp=0x10c000109b08 sp=0x10c000109ae8 pc=0x51969e

os/exec.(*Cmd).Output(0x10c000012340)

/root/godev/src/os/exec/exec.go:546 +0x245 fp=0x10c000109b98 sp=0x10c000109b08 pc=0x51b405

misc/cgo/testsanitizers_test.goEnv.func1()

/root/godev/misc/cgo/testsanitizers/cc_test.go:63 +0x6f fp=0x10c000109c20 sp=0x10c000109b98 pc=0x594fcf

sync.(*Once).doSlow(0x6ef640, 0x5e6230)

/root/godev/src/sync/once.go:68 +0xcf fp=0x10c000109c80 sp=0x10c000109c20 pc=0x47118f

sync.(*Once).Do(...)

/root/godev/src/sync/once.go:59

misc/cgo/testsanitizers_test.goEnv({0x5d9aeb, 0x4})

/root/godev/misc/cgo/testsanitizers/cc_test.go:61 +0x3f fp=0x10c000109cd8 sp=0x10c000109c80 pc=0x59517f

misc/cgo/testsanitizers_test.TestASAN(0x10c00010af00)

/root/godev/misc/cgo/testsanitizers/asan_test.go:13 +0x45 fp=0x10c000109f38 sp=0x10c000109cd8 pc=0x593da5

testing.tRunner(0x10c00010af00, 0x5e6208)

/root/godev/src/testing/testing.go:1440 +0x19e fp=0x10c000109fc0 sp=0x10c000109f38 pc=0x507abe

testing.(*T).Run.func1()

/root/godev/src/testing/testing.go:1487 +0x2a fp=0x10c000109fe0 sp=0x10c000109fc0 pc=0x50938a

runtime.goexit()

/root/godev/src/runtime/asm_amd64.s:1571 +0x1 fp=0x10c000109fe8 sp=0x10c000109fe0 pc=0x468881

created by testing.(*T).Run

/root/godev/src/testing/testing.go:1487 +0x777
goroutine 1 [chan receive]:

testing.(*T).Run(0x10c00010ad20, {0x5da571, 0x8}, 0x5e6208)

/root/godev/src/testing/testing.go:1488 +0x7a7

testing.runTests.func1(0x7f3ca2c35f60?)

/root/godev/src/testing/testing.go:1840 +0x95

testing.tRunner(0x10c00010ad20, 0x10c000107c18)

/root/godev/src/testing/testing.go:1440 +0x19e

testing.runTests(0x10c0001182a0?, {0x6d5400, 0x4, 0x4}, {0x10c000107d10?, 0x4ed6c8?, 0x0?})

/root/godev/src/testing/testing.go:1838 +0x83f

testing.(*M).Run(0x10c0001182a0)

/root/godev/src/testing/testing.go:1720 +0xb37

main.main()

_testmain.go:53 +0x2fd

fatal: morestack on g0

panic during panic

SIGTRAP: trace trap

PC=0x4686e2 m=0 sigcode=128
goroutine 0 [idle]:

runtime.abort()

/root/godev/src/runtime/asm_amd64.s:1036 +0x2

runtime.morestack()

/root/godev/src/runtime/asm_amd64.s:515 +0x25
goroutine 6 [running]:

runtime.asanwrite(0x10c0001091c0?, 0x20?)

/root/godev/src/runtime/asan.go:35 +0x5c fp=0x10c000109140 sp=0x10c000109138 pc=0x405a1c

runtime.sigaction(0x1, 0x10c0001091c0, 0x0)

/root/godev/src/runtime/cgo_sigaction.go:31 +0x32 fp=0x10c0001091a8 sp=0x10c000109140 pc=0x405db2

runtime.setsig(0x6d95c8?, 0x0?)

/root/godev/src/runtime/os_linux.go:468 +0x76 fp=0x10c0001091f0 sp=0x10c0001091a8 pc=0x4342f6

runtime.clearSignalHandlers()

/root/godev/src/runtime/signal_unix.go:261 +0x35 fp=0x10c000109218 sp=0x10c0001091f0 pc=0x44bd95

syscall.runtime_AfterForkInChild()

/root/godev/src/runtime/proc.go:4186 +0x1a fp=0x10c000109250 sp=0x10c000109218 pc=0x4641ba

syscall.forkAndExecInChild1(0x10c000018210, {0x10c00000e2a0, 0x4, 0x80000?}, {0x10c00012c3c0, 0x45817b, 0x2?}, 0x5de48b, 0x17, 0x10c0001097a0, ...)

/root/godev/src/syscall/exec_linux.go:217 +0x3ed fp=0x10c000109488 sp=0x10c000109250 pc=0x47456d

syscall.forkAndExecInChild(0x10c0000535a8?, {0x10c00000e2a0?, 0x7f3ca2c35ed8?, 0x340?}, {0x10c00012c3c0?, 0x440ec5?, 0x10c00010b0e0?}, 0x300000002?, 0x10c00010b0e0?, 0x71ead0, ...)

/root/godev/src/syscall/exec_linux.go:86 +0x98 fp=0x10c000109548 sp=0x10c000109488 pc=0x473ff8

syscall.forkExec({0x10c000018180, 0x14}, {0x10c00007e540, 0x3, 0x3}, 0x10c000012680?)

/root/godev/src/syscall/exec_unix.go:216 +0x53d fp=0x10c0001096c8 sp=0x10c000109548 pc=0x476add

syscall.StartProcess(...)

/root/godev/src/syscall/exec_unix.go:255

os.startProcess({0x10c000018180, 0x14}, {0x10c00007e540, 0x3, 0x3}, 0x10c000109a90)

/root/godev/src/os/exec_posix.go:54 +0x5cd fp=0x10c0001097f8 sp=0x10c0001096c8 pc=0x49c7cd

os.StartProcess({0x10c000018180, 0x14}, {0x10c00007e540, 0x3, 0x3}, 0x10c0000181b0?)

/root/godev/src/os/exec.go:109 +0x5a fp=0x10c000109840 sp=0x10c0001097f8 pc=0x49c11a

os/exec.(*Cmd).Start(0x10c000012340)

/root/godev/src/os/exec/exec.go:422 +0xa1d fp=0x10c000109ae8 sp=0x10c000109840 pc=0x51a0fd

os/exec.(*Cmd).Run(0x518265?)

/root/godev/src/os/exec/exec.go:338 +0x1e fp=0x10c000109b08 sp=0x10c000109ae8 pc=0x51969e

os/exec.(*Cmd).Output(0x10c000012340)

/root/godev/src/os/exec/exec.go:546 +0x245 fp=0x10c000109b98 sp=0x10c000109b08 pc=0x51b405

misc/cgo/testsanitizers_test.goEnv.func1()

/root/godev/misc/cgo/testsanitizers/cc_test.go:63 +0x6f fp=0x10c000109c20 sp=0x10c000109b98 pc=0x594fcf

sync.(*Once).doSlow(0x6ef640, 0x5e6230)

/root/godev/src/sync/once.go:68 +0xcf fp=0x10c000109c80 sp=0x10c000109c20 pc=0x47118f

sync.(*Once).Do(...)

/root/godev/src/sync/once.go:59

misc/cgo/testsanitizers_test.goEnv({0x5d9aeb, 0x4})

/root/godev/misc/cgo/testsanitizers/cc_test.go:61 +0x3f fp=0x10c000109cd8 sp=0x10c000109c80 pc=0x59517f

misc/cgo/testsanitizers_test.TestASAN(0x10c00010af00)

/root/godev/misc/cgo/testsanitizers/asan_test.go:13 +0x45 fp=0x10c000109f38 sp=0x10c000109cd8 pc=0x593da5

testing.tRunner(0x10c00010af00, 0x5e6208)

/root/godev/src/testing/testing.go:1440 +0x19e fp=0x10c000109fc0 sp=0x10c000109f38 pc=0x507abe

testing.(*T).Run.func1()

/root/godev/src/testing/testing.go:1487 +0x2a fp=0x10c000109fe0 sp=0x10c000109fc0 pc=0x50938a

runtime.goexit()

/root/godev/src/runtime/asm_amd64.s:1571 +0x1 fp=0x10c000109fe8 sp=0x10c000109fe0 pc=0x468881

created by testing.(*T).Run

/root/godev/src/testing/testing.go:1487 +0x777
goroutine 1 [chan receive]:

testing.(*T).Run(0x10c00010ad20, {0x5da571, 0x8}, 0x5e6208)

/root/godev/src/testing/testing.go:1488 +0x7a7

testing.runTests.func1(0x7f3ca2c35f60?)

/root/godev/src/testing/testing.go:1840 +0x95

testing.tRunner(0x10c00010ad20, 0x10c000107c18)

/root/godev/src/testing/testing.go:1440 +0x19e

testing.runTests(0x10c0001182a0?, {0x6d5400, 0x4, 0x4}, {0x10c000107d10?, 0x4ed6c8?, 0x0?})

/root/godev/src/testing/testing.go:1838 +0x83f

testing.(*M).Run(0x10c0001182a0)

/root/godev/src/testing/testing.go:1720 +0xb37

main.main()

_testmain.go:53 +0x2fd
rax    0x17

rbx    0x5de48b

rcx    0x46a095

rdx    0x17

rdi    0x2

rsi    0x5de48b

rbp    0x10c0001096b8

rsp    0x10c000109530

r8     0x0

r9     0x0

r10    0x8

r11    0x246

r12    0x47456d

r13    0x6f00c0

r14    0x6efb20

r15    0xffffffffffffffff

rip    0x4686e2

rflags 0x206

cs     0x33

fs     0x0

gs     0x0

kindly cc @zhangfannie @cherrymui

The text was updated successfully, but these errors were encountered:

zhangfannie · 2021-12-30T03:08:11Z

@mengzhuo Thank you for reporting this issue. The reason is that asanwrite/asanread function should be NOSPLIT. The fixed patch will be submitted soon. Thank you.

gopherbot · 2021-12-30T06:33:58Z

Change https://golang.org/cl/374398 mentions this issue: src/runtime: mark asanread and asanwrite functions as NOSPLIT

ALTree · 2021-12-30T09:20:05Z

Putting this in the 1.18 milestone because it looks like -asan will be shipped in 1.18, feel free to re-assess if this is not right.

The asan runtime functions may run on stacks that cannot grow, and they do not have large local variables, so it is safe to mark them as NOSPLIT. Add test case. Fixes golang#50391 Change-Id: Iadcbf1ae0c837d9b64da5be208c7f424e6ba11de Reviewed-on: https://go-review.googlesource.com/c/go/+/374398 Trust: Emmanuel Odeke <emmanuel@orijtech.com> Trust: Fannie Zhang <Fannie.Zhang@arm.com> Reviewed-by: Cherry Mui <cherryyz@google.com>

ALTree added this to the Go1.18 milestone Dec 30, 2021

ALTree added release-blocker NeedsFix labels Dec 30, 2021

gopherbot closed this as completed in 2c58bb2 Jan 5, 2022

thanm mentioned this issue Jan 25, 2022

Go compiler and runtime meeting notes #43930

Open

runtime: panic while TestASAN with "stack growth after fork" #50391

runtime: panic while TestASAN with "stack growth after fork" #50391

mengzhuo commented Dec 30, 2021 •

edited

zhangfannie commented Dec 30, 2021

gopherbot commented Dec 30, 2021

ALTree commented Dec 30, 2021

runtime: panic while TestASAN with "stack growth after fork" #50391

runtime: panic while TestASAN with "stack growth after fork" #50391

Comments

mengzhuo commented Dec 30, 2021 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

zhangfannie commented Dec 30, 2021

gopherbot commented Dec 30, 2021

ALTree commented Dec 30, 2021

mengzhuo commented Dec 30, 2021 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?