runtime: mcall called on m->g0 stack

[grant-singleton-nz]

What version of Go are you using (go version)?

$ go version
 go1.19.3 windows/amd64

Does this issue reproduce with the latest release?

1.19.3 is the latest release

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\gsingleton\AppData\Local\go-build
set GOENV=C:\Users\gsingleton\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\gsingleton\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\gsingleton\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.19.3
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=NUL
set GOWORK=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\GSINGL~1\AppData\Local\Temp\2\go-build411024941=/tmp/go-build -gno-record-gcc-switches

What did you do?

Cloned https://github.com/go-training/helloworld.git then

go run main.go

What did you expect to see?

Hello World!!

What did you see instead?

C:\helloworld>go run main.go
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x147d50a?, 0x160?})
c:/go/src/runtime/panic.go:1047 +0x65 fp=0xc000115a08 sp=0xc0001159d8 pc=0x12d68c5
runtime.badmcall(0x15eb710b4f8?)
c:/go/src/runtime/proc.go:468 +0x25 fp=0xc000115a28 sp=0xc000115a08 pc=0x12d9b45
runtime.badmcall(0x160)
:1 +0x2c fp=0xc000115a40 sp=0xc000115a28 pc=0x1302bcc
runtime.asyncPreempt2()
c:/go/src/runtime/preempt.go:308 +0x3f fp=0xc000115a60 sp=0xc000115a40 pc=0x12d7aff
runtime.asyncPreempt()
c:/go/src/runtime/preempt_amd64.s:50 +0xdb fp=0xc000115be8 sp=0xc000115a60 pc=0x1301f9b
cmd/internal/obj/arm.init()
c:/go/src/cmd/internal/obj/arm/a.out.go:113 fp=0xc000115bf0 sp=0xc000115be8 pc=0x13850c0
runtime.doInit(0x15a31a0)
c:/go/src/runtime/proc.go:6329 +0x12d fp=0xc000115d20 sp=0xc000115bf0 pc=0x12e5b6d
runtime.doInit(0x15a3960)
c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000115e50 sp=0xc000115d20 pc=0x12e5ab1
runtime.doInit(0x15a37a0)
c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000115f80 sp=0xc000115e50 pc=0x12e5ab1
runtime.main()
c:/go/src/runtime/proc.go:233 +0x1bf fp=0xc000115fe0 sp=0xc000115f80 pc=0x12d8fbf
runtime.goexit()
c:/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000115fe8 sp=0xc000115fe0 pc=0x13009a1

goroutine 1 [runnable, locked to thread]:
cmd/internal/obj/arm.init()
c:/go/src/cmd/internal/obj/arm/a.out.go:113 +0x11e fp=0xc000115bf0 sp=0xc000115be8 pc=0x13851de
runtime.doInit(0x15a31a0)
c:/go/src/runtime/proc.go:6329 +0x12d fp=0xc000115d20 sp=0xc000115bf0 pc=0x12e5b6d
runtime.doInit(0x15a3960)
c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000115e50 sp=0xc000115d20 pc=0x12e5ab1
runtime.doInit(0x15a37a0)
c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000115f80 sp=0xc000115e50 pc=0x12e5ab1
runtime.main()
c:/go/src/runtime/proc.go:233 +0x1bf fp=0xc000115fe0 sp=0xc000115f80 pc=0x12d8fbf
runtime.goexit()
c:/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000115fe8 sp=0xc000115fe0 pc=0x13009a1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
c:/go/src/runtime/proc.go:363 +0xd6 fp=0xc000053fb0 sp=0xc000053f90 pc=0x12d9396
runtime.goparkunlock(...)
c:/go/src/runtime/proc.go:369
runtime.forcegchelper()
c:/go/src/runtime/proc.go:302 +0xb1 fp=0xc000053fe0 sp=0xc000053fb0 pc=0x12d9231
runtime.goexit()
c:/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000053fe8 sp=0xc000053fe0 pc=0x13009a1
created by runtime.init.6
c:/go/src/runtime/proc.go:290 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
c:/go/src/runtime/proc.go:363 +0xd6 fp=0xc000055f90 sp=0xc000055f70 pc=0x12d9396
runtime.goparkunlock(...)
c:/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
c:/go/src/runtime/mgcsweep.go:278 +0x8e fp=0xc000055fc8 sp=0xc000055f90 pc=0x12c3fce
runtime.gcenable.func1()
c:/go/src/runtime/mgc.go:178 +0x26 fp=0xc000055fe0 sp=0xc000055fc8 pc=0x12b8d66
runtime.goexit()
c:/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000055fe8 sp=0xc000055fe0 pc=0x13009a1
created by runtime.gcenable
c:/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc00001c070?, 0x14c1488?, 0x1?, 0x0?, 0x0?)
c:/go/src/runtime/proc.go:363 +0xd6 fp=0xc000063f70 sp=0xc000063f50 pc=0x12d9396
runtime.goparkunlock(...)
c:/go/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x15fade0)
c:/go/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc000063fa0 sp=0xc000063f70 pc=0x12c2053
runtime.bgscavenge(0x0?)
c:/go/src/runtime/mgcscavenge.go:617 +0x45 fp=0xc000063fc8 sp=0xc000063fa0 pc=0x12c2645
runtime.gcenable.func2()
c:/go/src/runtime/mgc.go:179 +0x26 fp=0xc000063fe0 sp=0xc000063fc8 pc=0x12b8d06
runtime.goexit()
c:/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x13009a1
created by runtime.gcenable
c:/go/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait]:
runtime.gopark(0x0?, 0x15a1578?, 0x8b?, 0x61?, 0xc000057f70?)
c:/go/src/runtime/proc.go:363 +0xd6 fp=0xc000057e28 sp=0xc000057e08 pc=0x12d9396
runtime.goparkunlock(...)
c:/go/src/runtime/proc.go:369
runtime.runfinq()
c:/go/src/runtime/mfinal.go:180 +0x10f fp=0xc000057fe0 sp=0xc000057e28 pc=0x12b7e6f
runtime.goexit()
c:/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000057fe8 sp=0xc000057fe0 pc=0x13009a1
created by runtime.createfing
c:/go/src/runtime/mfinal.go:157 +0x45
go: error obtaining buildID for go tool asm: exit status 2

[joedian]

@grant-singleton-nz can you please update the title of this issue to be more descriptive?

[seankhliao]

Does this crash on running go build ?
Do other go programs crash?
What can you tell us about the OS?

[grant-singleton-nz]

Does this crash on running go build ?

C:\helloworld>go build
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x16faba3?, 0x0?})
        c:/go/src/runtime/panic.go:1047 +0x65 fp=0xc000122d30 sp=0xc000122d00 pc=0x10e8ba5
runtime.badmcall(0x0?)
        c:/go/src/runtime/proc.go:468 +0x25 fp=0xc000122d50 sp=0xc000122d30 pc=0x10ebe25
runtime.badmcall(0x0)
        <autogenerated>:1 +0x2c fp=0xc000122d68 sp=0xc000122d50 pc=0x1118aac
runtime.asyncPreempt2()
        c:/go/src/runtime/preempt.go:308 +0x3f fp=0xc000122d88 sp=0xc000122d68 pc=0x10e9ddf
runtime.asyncPreempt()
        c:/go/src/runtime/preempt_amd64.s:50 +0xdb fp=0xc000122f10 sp=0xc000122d88 pc=0x1117dfb
crypto/internal/nistec/fiat.p384FromMontgomery(0xc000122f28, 0xc000123110)
        c:/go/src/crypto/internal/nistec/fiat/p384_fiat64.go:1644 fp=0xc000122f18 sp=0xc000122f10 pc=0x12a1d40
crypto/internal/nistec/fiat.(*P384Element).bytes(0xc000122fa8?, 0xc00001e660)
        c:/go/src/crypto/internal/nistec/fiat/p384.go:64 +0x45 fp=0xc000122f68 sp=0xc000122f18 pc=0x129d685
crypto/internal/nistec/fiat.(*P384Element).Bytes(...)
        c:/go/src/crypto/internal/nistec/fiat/p384.go:59
crypto/internal/nistec/fiat.init()
        c:/go/src/crypto/internal/nistec/fiat/p384.go:40 +0x285 fp=0xc000123270 sp=0xc000122f68 pc=0x12b42c5
runtime.doInit(0x1aae420)
        c:/go/src/runtime/proc.go:6329 +0x12d fp=0xc0001233a0 sp=0xc000123270 pc=0x10f7e4d
runtime.doInit(0x1ab2480)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc0001234d0 sp=0xc0001233a0 pc=0x10f7d91
runtime.doInit(0x1ab0020)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123600 sp=0xc0001234d0 pc=0x10f7d91
runtime.doInit(0x1ab3ae0)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123730 sp=0xc000123600 pc=0x10f7d91
runtime.doInit(0x1ab7da0)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123860 sp=0xc000123730 pc=0x10f7d91
runtime.doInit(0x1ab54c0)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123990 sp=0xc000123860 pc=0x10f7d91
runtime.doInit(0x1ab62a0)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123ac0 sp=0xc000123990 pc=0x10f7d91
runtime.doInit(0x1ab7f00)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123bf0 sp=0xc000123ac0 pc=0x10f7d91
runtime.doInit(0x1ab8600)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123d20 sp=0xc000123bf0 pc=0x10f7d91
runtime.doInit(0x1ab5580)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123e50 sp=0xc000123d20 pc=0x10f7d91
runtime.doInit(0x1ab74c0)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123f80 sp=0xc000123e50 pc=0x10f7d91
runtime.main()
        c:/go/src/runtime/proc.go:233 +0x1bf fp=0xc000123fe0 sp=0xc000123f80 pc=0x10eb29f
runtime.goexit()
        c:/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000123fe8 sp=0xc000123fe0 pc=0x11167c1

goroutine 1 [runnable, locked to thread]:
crypto/internal/nistec/fiat.p384FromMontgomery(0xc000122f28, 0xc000123110)
        c:/go/src/crypto/internal/nistec/fiat/p384_fiat64.go:1644 +0x131a fp=0xc000122f18 sp=0xc000122f10 pc=0x12a305a
crypto/internal/nistec/fiat.(*P384Element).bytes(0xc000122fa8?, 0xc00001e660)
        c:/go/src/crypto/internal/nistec/fiat/p384.go:64 +0x45 fp=0xc000122f68 sp=0xc000122f18 pc=0x129d685
crypto/internal/nistec/fiat.(*P384Element).Bytes(...)
        c:/go/src/crypto/internal/nistec/fiat/p384.go:59
crypto/internal/nistec/fiat.init()
        c:/go/src/crypto/internal/nistec/fiat/p384.go:40 +0x285 fp=0xc000123270 sp=0xc000122f68 pc=0x12b42c5
runtime.doInit(0x1aae420)
        c:/go/src/runtime/proc.go:6329 +0x12d fp=0xc0001233a0 sp=0xc000123270 pc=0x10f7e4d
runtime.doInit(0x1ab2480)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc0001234d0 sp=0xc0001233a0 pc=0x10f7d91
runtime.doInit(0x1ab0020)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123600 sp=0xc0001234d0 pc=0x10f7d91
runtime.doInit(0x1ab3ae0)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123730 sp=0xc000123600 pc=0x10f7d91
runtime.doInit(0x1ab7da0)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123860 sp=0xc000123730 pc=0x10f7d91
runtime.doInit(0x1ab54c0)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123990 sp=0xc000123860 pc=0x10f7d91
runtime.doInit(0x1ab62a0)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123ac0 sp=0xc000123990 pc=0x10f7d91
runtime.doInit(0x1ab7f00)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123bf0 sp=0xc000123ac0 pc=0x10f7d91
runtime.doInit(0x1ab8600)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123d20 sp=0xc000123bf0 pc=0x10f7d91
runtime.doInit(0x1ab5580)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123e50 sp=0xc000123d20 pc=0x10f7d91
runtime.doInit(0x1ab74c0)
        c:/go/src/runtime/proc.go:6306 +0x71 fp=0xc000123f80 sp=0xc000123e50 pc=0x10f7d91
runtime.main()
        c:/go/src/runtime/proc.go:233 +0x1bf fp=0xc000123fe0 sp=0xc000123f80 pc=0x10eb29f
runtime.goexit()
        c:/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000123fe8 sp=0xc000123fe0 pc=0x11167c1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        c:/go/src/runtime/proc.go:363 +0xd6 fp=0xc000061fb0 sp=0xc000061f90 pc=0x10eb676
runtime.goparkunlock(...)
        c:/go/src/runtime/proc.go:369
runtime.forcegchelper()
        c:/go/src/runtime/proc.go:302 +0xb1 fp=0xc000061fe0 sp=0xc000061fb0 pc=0x10eb511
runtime.goexit()
        c:/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x11167c1
created by runtime.init.6
        c:/go/src/runtime/proc.go:290 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        c:/go/src/runtime/proc.go:363 +0xd6 fp=0xc000063f90 sp=0xc000063f70 pc=0x10eb676
runtime.goparkunlock(...)
        c:/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
        c:/go/src/runtime/mgcsweep.go:278 +0x8e fp=0xc000063fc8 sp=0xc000063f90 pc=0x10d580e
runtime.gcenable.func1()
        c:/go/src/runtime/mgc.go:178 +0x26 fp=0xc000063fe0 sp=0xc000063fc8 pc=0x10ca3a6
runtime.goexit()
        c:/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x11167c1
created by runtime.gcenable
        c:/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc00002a070?, 0x17e8180?, 0x1?, 0x0?, 0x0?)
        c:/go/src/runtime/proc.go:363 +0xd6 fp=0xc000073f70 sp=0xc000073f50 pc=0x10eb676
runtime.goparkunlock(...)
        c:/go/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x1afa420)
        c:/go/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc000073fa0 sp=0xc000073f70 pc=0x10d3893
runtime.bgscavenge(0x0?)
        c:/go/src/runtime/mgcscavenge.go:617 +0x45 fp=0xc000073fc8 sp=0xc000073fa0 pc=0x10d3e85
runtime.gcenable.func2()
        c:/go/src/runtime/mgc.go:179 +0x26 fp=0xc000073fe0 sp=0xc000073fc8 pc=0x10ca346
runtime.goexit()
        c:/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000073fe8 sp=0xc000073fe0 pc=0x11167c1
created by runtime.gcenable
        c:/go/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait]:
runtime.gopark(0x0?, 0xc000065e70?, 0xcb?, 0x77?, 0xc000065f70?)
        c:/go/src/runtime/proc.go:363 +0xd6 fp=0xc000065e28 sp=0xc000065e08 pc=0x10eb676
runtime.goparkunlock(...)
        c:/go/src/runtime/proc.go:369
runtime.runfinq()
        c:/go/src/runtime/mfinal.go:180 +0x10f fp=0xc000065fe0 sp=0xc000065e28 pc=0x10c94af
runtime.goexit()
        c:/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000065fe8 sp=0xc000065fe0 pc=0x11167c1
created by runtime.createfing
        c:/go/src/runtime/mfinal.go:157 +0x45

Do other go programs crash?

Yes, I think every go program I've tried does this. Its not always the same stack dump either, and sometimes it'll run as expected. Sounds like stack corruption? I can post more stack dumps if that will help.

What can you tell us about the OS?

OS is Windows Server 2019. Its a VM and the hypervisor is Bhyve, host OS is SmartOS

[grant-singleton-nz]

I've discovered a little bit more information - don't know if it'll be useful

We have seven host machines with SmartOS running various VMs and this issue only occurs on the ones that have AMD Epyc 7543 CPUs. This is the CPUs we have got and the problem doesn't occur on the Intels or the 7542s

2x AMD EPYC 7543 32-Core
2x AMD EPYC 7542 32-Core
1x Intel(R) Xeon(R) CPU E5-2687W v4 @ 3.00GHz
2x Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz

[cherrymui]

OS is Windows Server 2019. Its a VM and the hypervisor is Bhyve, host OS is SmartOS

Is it reproducible on a physical Windows machine? From the stack trace it looks to me something is wrong with thread local storage. I wonder if the VM handles it correctly?

I'm not sure this is the right place to look of if it is up-to-date, but https://wiki.freebsd.org/bhyve/Windows mentions

bhyve supports Windows desktop versions Vista, 7, 8/8.1/8.2, 10 and 11, as well as Windows Server versions 2008/2008R2, 2012/2012R2 and 2016 Technical Preview 2 and 3

It doesn't include Windows Server 2019. So maybe Windows Server 2019 is not well supported?

Thanks.

[grant-singleton-nz]

I've never seen this happen on a physical machine - but we only have windows 10 on physical machines, the windows server instances are all VMs. We also don't have any windows 10 VMs running on SmartOS/bhyve.

I done some more testing and it does happen on the AMD 7542's just not as frequently. So this is looking like something to do with SmartOS and/or bhyve with a windows VM on an AMD processor

I've raised a ticket with the SmartOS people https://smartos.topicbox.com/groups/smartos-discuss/T6938eca11dfeb1e3/golang-executables-crash-in-windows-vms-on-amd, will see what they say

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[grant-singleton-nz]

Seems this is resolved in 1.21.3. Today I tried 1.18.10 and the problem exists there. Then I upgraded to 1.21.3 and its working perfectly.