Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: TestG0StackOverflow timeout on windows/arm64 #63938

Closed
cherrymui opened this issue Nov 3, 2023 · 13 comments
Closed

runtime: TestG0StackOverflow timeout on windows/arm64 #63938

cherrymui opened this issue Nov 3, 2023 · 13 comments
Assignees
Labels
arch-arm64 compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows release-blocker
Milestone

Comments

@cherrymui
Copy link
Member

TestG0StackOverflow timeout on windows/arm64 since CL https://go.dev/cl/538457 .

I'll look into it.

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Nov 3, 2023
@cherrymui cherrymui self-assigned this Nov 3, 2023
@cherrymui cherrymui added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 3, 2023
@cherrymui cherrymui added this to the Go1.22 milestone Nov 3, 2023
@heschi
Copy link
Contributor

heschi commented Nov 9, 2023

Ping. Can we get this fixed or skipped or something?

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/541195 mentions this issue: runtime: skip TestG0StackOverflow on windows/arm64

gopherbot pushed a commit that referenced this issue Nov 9, 2023
Temporarily skip to make the builder happy. Will work on a fix.

Updates #63938.

Change-Id: Ic9db771342108430c29774b2c3e50043791189a6
Reviewed-on: https://go-review.googlesource.com/c/go/+/541195
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Cherry Mui <cherryyz@google.com>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/541997 mentions this issue: runtime: allocate crash stack in BSS on windows

@cherrymui
Copy link
Member Author

Tried to reenable this with some additional logging. When it fails, the log is like below. Specifically, it first prints the stack traces as expected, then when it about to crash the program at the end, it gets Exception 0xc00000fd (EXCEPTION_STACK_OVERFLOW), on the crash stack. The exception triggers another round of stack dumps, and somehow hangs...

Using a static allocation for the crash stack (like CL https://go.dev/cl/541997 does), it doesn't cause the exception.

When the runtime finally crashes from a throw, it calls crash, which calls dieFromExcpetion on Windows, which throws an exception https://cs.opensource.google/go/go/+/master:src/runtime/signal_windows.go;l=448 . I guess this throwing exception actually causes an EXCEPTION_STACK_OVERFLOW, although it appears we still have plenty of space on the crash stack (tried bumping up the crash size to quite large and it doesn't help, also the faulting SP is pretty close to the stack top). I wonder if Windows/ARM64 requires an exception to only be thrown on a system allocated stack? MSDN search found something related to guard page, but it doesn't seem to clearly suggest that the stack must be system allocated or needs something special...

@qmuntal do you have any insight to share? Thanks!

log

=== RUN   TestG0StackOverflow
    exec.go:193: test timed out while running command: C:\workdir\runtime.test.exe -test.run=^TestG0StackOverflow$ -test.v
    crash_test.go:803: output:
        === RUN   TestG0StackOverflow
        runtime: morestack on g0, stack [0xd9483fdf20 0xd9483fff60], sp=0xd9483ff2a0, called from
        runtime.stackOverflow(0x0?)
        	/Users/cherryyz/src/go/src/runtime/export_test.go:729 +0x54 fp=0xd9483ff2a0 sp=0xd9483ff2a0 pc=0x7ff76d5dcef4
        runtime.stackOverflow(0x0?)
        	/Users/cherryyz/src/go/src/runtime/export_test.go:731 +0x40 fp=0xd9483ff3c0 sp=0xd9483ff2a0 pc=0x7ff76d5dcee0
        runtime.stackOverflow(0x0?)
        	/Users/cherryyz/src/go/src/runtime/export_test.go:731 +0x40 fp=0xd9483ff4e0 sp=0xd9483ff3c0 pc=0x7ff76d5dcee0
        runtime.stackOverflow(0x259bfa0?)
        	/Users/cherryyz/src/go/src/runtime/export_test.go:731 +0x40 fp=0xd9483ff600 sp=0xd9483ff4e0 pc=0x7ff76d5dcee0
        runtime.stackOverflow(0x23ccf7b0000?)
        	/Users/cherryyz/src/go/src/runtime/export_test.go:731 +0x40 fp=0xd9483ff720 sp=0xd9483ff600 pc=0x7ff76d5dcee0
        runtime.stackOverflow(0xa6?)
        	/Users/cherryyz/src/go/src/runtime/export_test.go:731 +0x40 fp=0xd9483ff840 sp=0xd9483ff720 pc=0x7ff76d5dcee0
        runtime.stackOverflow(0x7ffce7aef424?)
        	/Users/cherryyz/src/go/src/runtime/export_test.go:731 +0x40 fp=0xd9483ff960 sp=0xd9483ff840 pc=0x7ff76d5dcee0
        runtime.stackOverflow(0x64006b0072006f?)
        	/Users/cherryyz/src/go/src/runtime/export_test.go:731 +0x40 fp=0xd9483ffa80 sp=0xd9483ff960 pc=0x7ff76d5dcee0
        runtime.stackOverflow(0xd9483ffc28?)
        	/Users/cherryyz/src/go/src/runtime/export_test.go:731 +0x40 fp=0xd9483ffba0 sp=0xd9483ffa80 pc=0x7ff76d5dcee0
        runtime.stackOverflow(0xd9483ffe90?)
        	/Users/cherryyz/src/go/src/runtime/export_test.go:731 +0x40 fp=0xd9483ffcc0 sp=0xd9483ffba0 pc=0x7ff76d5dcee0
        runtime.stackOverflow(0x0?)
        	/Users/cherryyz/src/go/src/runtime/export_test.go:731 +0x40 fp=0xd9483ffde0 sp=0xd9483ffcc0 pc=0x7ff76d5dcee0
        runtime.stackOverflow(0x4000061e40?)
        	/Users/cherryyz/src/go/src/runtime/export_test.go:731 +0x40 fp=0xd9483fff00 sp=0xd9483ffde0 pc=0x7ff76d5dcee0
        runtime_test.TestG0StackOverflow.G0StackOverflow.func1()
        	/Users/cherryyz/src/go/src/runtime/export_test.go:725 +0x40 fp=0xd9483fff20 sp=0xd9483fff00 pc=0x7ff76d8da200
        runtime.systemstack(0x8)
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:243 +0x6c fp=0xd9483fff30 sp=0xd9483fff20 pc=0x7ff76d5e8fcc
        
        fatal error: morestack on g0
        
        runtime stack:
        runtime.throw({0x7ff76d99f848?, 0x0?})
        	/Users/cherryyz/src/go/src/runtime/panic.go:1016 +0x40 fp=0x4000037f80 sp=0x4000037f50 pc=0x7ff76d5adef0
        runtime.badmorestackg0.func1()
        	/Users/cherryyz/src/go/src/runtime/proc.go:533 +0xcc fp=0x4000037fe0 sp=0x4000037f80 pc=0x7ff76d5b21bc
        runtime.switchToCrashStack0()
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:283 +0x3c fp=0x4000037ff0 sp=0x4000037fe0 pc=0x7ff76d5e904c
        
        goroutine 6 gp=0x40000496c0 m=0 [running]:
        runtime.systemstack_switch()
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:200 +0x8 fp=0x4000061e40 sp=0x4000061e30 pc=0x7ff76d5e8f48
        runtime.G0StackOverflow(...)
        	/Users/cherryyz/src/go/src/runtime/export_test.go:715
        runtime_test.TestG0StackOverflow(0x4000049520)
        	/Users/cherryyz/src/go/src/runtime/crash_test.go:825 +0x2f8 fp=0x4000061f60 sp=0x4000061e40 pc=0x7ff76d845778
        testing.tRunner(0x4000049520, 0x7ff76d9c7a58)
        	/Users/cherryyz/src/go/src/testing/testing.go:1682 +0xec fp=0x4000061fb0 sp=0x4000061f60 pc=0x7ff76d689fac
        testing.(*T).Run.gowrap1()
        	/Users/cherryyz/src/go/src/testing/testing.go:1735 +0x2c fp=0x4000061fd0 sp=0x4000061fb0 pc=0x7ff76d68ad7c
        runtime.goexit({})
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x4000061fd0 sp=0x4000061fd0 pc=0x7ff76d5eb444
        created by testing.(*T).Run in goroutine 1
        	/Users/cherryyz/src/go/src/testing/testing.go:1735 +0x318
        
        goroutine 1 gp=0x4000048000 m=nil [chan receive]:
        runtime.gopark(0x400007f918?, 0x400002add0?, 0x58?, 0xae?, 0x400002add0?)
        	/Users/cherryyz/src/go/src/runtime/proc.go:402 +0xc8 fp=0x400007f8e0 sp=0x400007f8c0 pc=0x7ff76d5b19f8
        runtime.chanrecv(0x40000262a0, 0x400007f9df, 0x1)
        	/Users/cherryyz/src/go/src/runtime/chan.go:583 +0x414 fp=0x400007f960 sp=0x400007f8e0 pc=0x7ff76d578244
        runtime.chanrecv1(0x7ff76dc98880?, 0x7ff76d90e800?)
        	/Users/cherryyz/src/go/src/runtime/chan.go:442 +0x14 fp=0x400007f990 sp=0x400007f960 pc=0x7ff76d577df4
        testing.(*T).Run(0x4000049380, {0x7ff76d9a2a7e?, 0x400007fa88?}, 0x7ff76d9c7a58)
        	/Users/cherryyz/src/go/src/testing/testing.go:1736 +0x32c fp=0x400007fa50 sp=0x400007f990 pc=0x7ff76d68ac4c
        testing.runTests.func1(0x4000049380)
        	/Users/cherryyz/src/go/src/testing/testing.go:2146 +0x40 fp=0x400007fa90 sp=0x400007fa50 pc=0x7ff76d68ca90
        testing.tRunner(0x4000049380, 0x400007fbb8)
        	/Users/cherryyz/src/go/src/testing/testing.go:1682 +0xec fp=0x400007fae0 sp=0x400007fa90 pc=0x7ff76d689fac
        testing.runTests(0x4000004108, {0x7ff76dc94f00?, 0x1a8, 0x1a8}, {0x400007fc38?, 0x7ff76d57f584?, 0x0?})
        	/Users/cherryyz/src/go/src/testing/testing.go:2144 +0x3b8 fp=0x400007fbe0 sp=0x400007fae0 pc=0x7ff76d68c988
        testing.(*M).Run(0x4000072820)
        	/Users/cherryyz/src/go/src/testing/testing.go:2012 +0x5a4 fp=0x400007fe10 sp=0x400007fbe0 pc=0x7ff76d68b6a4
        runtime_test.TestMain(0x4000072820)
        	/Users/cherryyz/src/go/src/runtime/crash_test.go:29 +0x38 fp=0x400007fe80 sp=0x400007fe10 pc=0x7ff76d840398
        main.main()
        	_testmain.go:1525 +0x170 fp=0x400007ff40 sp=0x400007fe80 pc=0x7ff76d8e14f0
        runtime.main()
        	/Users/cherryyz/src/go/src/runtime/proc.go:271 +0x274 fp=0x400007ffd0 sp=0x400007ff40 pc=0x7ff76d5b15c4
        runtime.goexit({})
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400007ffd0 sp=0x400007ffd0 pc=0x7ff76d5eb444
        
        goroutine 2 gp=0x4000048680 m=nil [force gc (idle)]:
        runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        	/Users/cherryyz/src/go/src/runtime/proc.go:402 +0xc8 fp=0x400004bf90 sp=0x400004bf70 pc=0x7ff76d5b19f8
        runtime.goparkunlock(...)
        	/Users/cherryyz/src/go/src/runtime/proc.go:408
        runtime.forcegchelper()
        	/Users/cherryyz/src/go/src/runtime/proc.go:326 +0xb8 fp=0x400004bfd0 sp=0x400004bf90 pc=0x7ff76d5b1858
        runtime.goexit({})
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400004bfd0 sp=0x400004bfd0 pc=0x7ff76d5eb444
        created by runtime.init.6 in goroutine 1
        	/Users/cherryyz/src/go/src/runtime/proc.go:314 +0x24
        
        goroutine 3 gp=0x40000489c0 m=nil [GC sweep wait]:
        runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        	/Users/cherryyz/src/go/src/runtime/proc.go:402 +0xc8 fp=0x400004df60 sp=0x400004df40 pc=0x7ff76d5b19f8
        runtime.goparkunlock(...)
        	/Users/cherryyz/src/go/src/runtime/proc.go:408
        runtime.bgsweep(0x4000026070)
        	/Users/cherryyz/src/go/src/runtime/mgcsweep.go:277 +0xa0 fp=0x400004dfb0 sp=0x400004df60 pc=0x7ff76d599a40
        runtime.gcenable.gowrap1()
        	/Users/cherryyz/src/go/src/runtime/mgc.go:203 +0x28 fp=0x400004dfd0 sp=0x400004dfb0 pc=0x7ff76d58dba8
        runtime.goexit({})
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400004dfd0 sp=0x400004dfd0 pc=0x7ff76d5eb444
        created by runtime.gcenable in goroutine 1
        	/Users/cherryyz/src/go/src/runtime/mgc.go:203 +0x6c
        
        goroutine 4 gp=0x4000048b60 m=nil [GC scavenge wait]:
        runtime.gopark(0x4000026070?, 0x7ff76da3f948?, 0x1?, 0x0?, 0x4000048b60?)
        	/Users/cherryyz/src/go/src/runtime/proc.go:402 +0xc8 fp=0x400005ff60 sp=0x400005ff40 pc=0x7ff76d5b19f8
        runtime.goparkunlock(...)
        	/Users/cherryyz/src/go/src/runtime/proc.go:408
        runtime.(*scavengerState).park(0x7ff76dc99100)
        	/Users/cherryyz/src/go/src/runtime/mgcscavenge.go:425 +0x5c fp=0x400005ff90 sp=0x400005ff60 pc=0x7ff76d59744c
        runtime.bgscavenge(0x4000026070)
        	/Users/cherryyz/src/go/src/runtime/mgcscavenge.go:653 +0x44 fp=0x400005ffb0 sp=0x400005ff90 pc=0x7ff76d597994
        runtime.gcenable.gowrap2()
        	/Users/cherryyz/src/go/src/runtime/mgc.go:204 +0x28 fp=0x400005ffd0 sp=0x400005ffb0 pc=0x7ff76d58db48
        runtime.goexit({})
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400005ffd0 sp=0x400005ffd0 pc=0x7ff76d5eb444
        created by runtime.gcenable in goroutine 1
        	/Users/cherryyz/src/go/src/runtime/mgc.go:204 +0xac
        
        goroutine 5 gp=0x4000048ea0 m=nil [finalizer wait]:
        runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        	/Users/cherryyz/src/go/src/runtime/proc.go:402 +0xc8 fp=0x400004fd80 sp=0x400004fd60 pc=0x7ff76d5b19f8
        runtime.runfinq()
        	/Users/cherryyz/src/go/src/runtime/mfinal.go:193 +0x108 fp=0x400004ffd0 sp=0x400004fd80 pc=0x7ff76d58cc38
        runtime.goexit({})
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400004ffd0 sp=0x400004ffd0 pc=0x7ff76d5eb444
        created by runtime.createfing in goroutine 1
        	/Users/cherryyz/src/go/src/runtime/mfinal.go:163 +0x80
        Exception 0xc00000fd 0x0 0xd948203000 0x7ffce7a62740
        PC=0x7ffce7a62740
        
        runtime: g 0 gp=0x7ff76dc99520: unknown pc 0x7ffce7a62740
        stack: frame={sp:0x4000037a50, fp:0x0} stack=[0x0,0x4000038000)
        0x0000004000037950:  0x0000000000000000  0x0000000000000000 
        0x0000004000037960:  0x0000000000000000  0x0000000000000000 
        0x0000004000037970:  0x0000000000000000  0x0000000000000000 
        0x0000004000037980:  0x0000000000000000  0x0000000000000000 
        0x0000004000037990:  0x00000008c00000fd  0x0000000000000000 
        0x00000040000379a0:  0x00007ffce7a62740  0x0000000000000002 
        0x00000040000379b0:  0x0000000000000000  0x000000d948203000 
        0x00000040000379c0:  0x0000000000000000  0x0000000000000000 
        0x00000040000379d0:  0x0000000000000000  0x0000000000000000 
        0x00000040000379e0:  0x0000000000000000  0x0000000000000000 
        0x00000040000379f0:  0x0000000000000000  0x0000000000000000 
        0x0000004000037a00:  0x0000000000000000  0x0000000000000000 
        0x0000004000037a10:  0x0000000000000000  0x0000000000000000 
        0x0000004000037a20:  0x0000000000000000  0x0000000000000001 
        0x0000004000037a30:  0x0000004000037a50  0x00007ffce7a62740 
        0x0000004000037a40:  0x0000023ccf815690  0x0000023ccf815690 
        0x0000004000037a50: <0x0000023ccf8156d0  0xffffacb61620656a 
        0x0000004000037a60:  0x0000004000037af0  0x00007ffce3343ce4 
        0x0000004000037a70:  0x0000004000037b10  0x00007ffce38a4670 
        0x0000004000037a80:  0x00007ffce33408b0  0x0000000000000000 
        0x0000004000037a90:  0x00007ffce3300000  0x0000000000000044 
        0x0000004000037aa0:  0x0000000000000001  0x00007ffce3316290 
        0x0000004000037ab0:  0x418a073aa3bc1c75  0x418a073aa3bc2475 
        0x0000004000037ac0:  0x418a073aa3bc2c75  0x00007ffce3345abc 
        0x0000004000037ad0:  0x0000023ccf8066b0  0x0000004000037b10 
        0x0000004000037ae0:  0x00007ffce33408b0  0xffffacb6162065fa 
        0x0000004000037af0:  0x0000004000037be0  0x00007ffce3345a74 
        0x0000004000037b00:  0x0000023ccf806690  0x00007ffce38a4670 
        0x0000004000037b10:  0x0004000100040000  0x00007ffce3316200 
        0x0000004000037b20:  0x0000000000000004  0x0000023ccf815690 
        0x0000004000037b30:  0x0000023ccf8156a4  0x0000023ccf8156d0 
        0x0000004000037b40:  0x0000023ccf815690  0x00007ffce7000001 
        runtime: g 0 gp=0x7ff76dc99520: unknown pc 0x7ffce7a62740
        stack: frame={sp:0x4000037a50, fp:0x0} stack=[0x0,0x4000038000)
        0x0000004000037950:  0x0000000000000000  0x0000000000000000 
        0x0000004000037960:  0x0000000000000000  0x0000000000000000 
        0x0000004000037970:  0x0000000000000000  0x0000000000000000 
        0x0000004000037980:  0x0000000000000000  0x0000000000000000 
        0x0000004000037990:  0x00000008c00000fd  0x0000000000000000 
        0x00000040000379a0:  0x00007ffce7a62740  0x0000000000000002 
        0x00000040000379b0:  0x0000000000000000  0x000000d948203000 
        0x00000040000379c0:  0x0000000000000000  0x0000000000000000 
        0x00000040000379d0:  0x0000000000000000  0x0000000000000000 
        0x00000040000379e0:  0x0000000000000000  0x0000000000000000 
        0x00000040000379f0:  0x0000000000000000  0x0000000000000000 
        0x0000004000037a00:  0x0000000000000000  0x0000000000000000 
        0x0000004000037a10:  0x0000000000000000  0x0000000000000000 
        0x0000004000037a20:  0x0000000000000000  0x0000000000000001 
        0x0000004000037a30:  0x0000004000037a50  0x00007ffce7a62740 
        0x0000004000037a40:  0x0000023ccf815690  0x0000023ccf815690 
        0x0000004000037a50: <0x0000023ccf8156d0  0xffffacb61620656a 
        0x0000004000037a60:  0x0000004000037af0  0x00007ffce3343ce4 
        0x0000004000037a70:  0x0000004000037b10  0x00007ffce38a4670 
        0x0000004000037a80:  0x00007ffce33408b0  0x0000000000000000 
        0x0000004000037a90:  0x00007ffce3300000  0x0000000000000044 
        0x0000004000037aa0:  0x0000000000000001  0x00007ffce3316290 
        0x0000004000037ab0:  0x418a073aa3bc1c75  0x418a073aa3bc2475 
        0x0000004000037ac0:  0x418a073aa3bc2c75  0x00007ffce3345abc 
        0x0000004000037ad0:  0x0000023ccf8066b0  0x0000004000037b10 
        0x0000004000037ae0:  0x00007ffce33408b0  0xffffacb6162065fa 
        0x0000004000037af0:  0x0000004000037be0  0x00007ffce3345a74 
        0x0000004000037b00:  0x0000023ccf806690  0x00007ffce38a4670 
        0x0000004000037b10:  0x0004000100040000  0x00007ffce3316200 
        0x0000004000037b20:  0x0000000000000004  0x0000023ccf815690 
        0x0000004000037b30:  0x0000023ccf8156a4  0x0000023ccf8156d0 
        0x0000004000037b40:  0x0000023ccf815690  0x00007ffce7000001 
        
        goroutine 6 gp=0x40000496c0 m=0 [running]:
        runtime.systemstack_switch()
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:200 +0x8 fp=0x4000061e40 sp=0x4000061e30 pc=0x7ff76d5e8f48
        runtime.G0StackOverflow(...)
        	/Users/cherryyz/src/go/src/runtime/export_test.go:715
        runtime_test.TestG0StackOverflow(0x4000049520)
        	/Users/cherryyz/src/go/src/runtime/crash_test.go:825 +0x2f8 fp=0x4000061f60 sp=0x4000061e40 pc=0x7ff76d845778
        testing.tRunner(0x4000049520, 0x7ff76d9c7a58)
        	/Users/cherryyz/src/go/src/testing/testing.go:1682 +0xec fp=0x4000061fb0 sp=0x4000061f60 pc=0x7ff76d689fac
        testing.(*T).Run.gowrap1()
        	/Users/cherryyz/src/go/src/testing/testing.go:1735 +0x2c fp=0x4000061fd0 sp=0x4000061fb0 pc=0x7ff76d68ad7c
        runtime.goexit({})
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x4000061fd0 sp=0x4000061fd0 pc=0x7ff76d5eb444
        created by testing.(*T).Run in goroutine 1
        	/Users/cherryyz/src/go/src/testing/testing.go:1735 +0x318
        
        goroutine 1 gp=0x4000048000 m=nil [chan receive]:
        runtime.gopark(0x400007f918?, 0x400002add0?, 0x58?, 0xae?, 0x400002add0?)
        	/Users/cherryyz/src/go/src/runtime/proc.go:402 +0xc8 fp=0x400007f8e0 sp=0x400007f8c0 pc=0x7ff76d5b19f8
        runtime.chanrecv(0x40000262a0, 0x400007f9df, 0x1)
        	/Users/cherryyz/src/go/src/runtime/chan.go:583 +0x414 fp=0x400007f960 sp=0x400007f8e0 pc=0x7ff76d578244
        runtime.chanrecv1(0x7ff76dc98880?, 0x7ff76d90e800?)
        	/Users/cherryyz/src/go/src/runtime/chan.go:442 +0x14 fp=0x400007f990 sp=0x400007f960 pc=0x7ff76d577df4
        testing.(*T).Run(0x4000049380, {0x7ff76d9a2a7e?, 0x400007fa88?}, 0x7ff76d9c7a58)
        	/Users/cherryyz/src/go/src/testing/testing.go:1736 +0x32c fp=0x400007fa50 sp=0x400007f990 pc=0x7ff76d68ac4c
        testing.runTests.func1(0x4000049380)
        	/Users/cherryyz/src/go/src/testing/testing.go:2146 +0x40 fp=0x400007fa90 sp=0x400007fa50 pc=0x7ff76d68ca90
        testing.tRunner(0x4000049380, 0x400007fbb8)
        	/Users/cherryyz/src/go/src/testing/testing.go:1682 +0xec fp=0x400007fae0 sp=0x400007fa90 pc=0x7ff76d689fac
        testing.runTests(0x4000004108, {0x7ff76dc94f00?, 0x1a8, 0x1a8}, {0x400007fc38?, 0x7ff76d57f584?, 0x0?})
        	/Users/cherryyz/src/go/src/testing/testing.go:2144 +0x3b8 fp=0x400007fbe0 sp=0x400007fae0 pc=0x7ff76d68c988
        testing.(*M).Run(0x4000072820)
        	/Users/cherryyz/src/go/src/testing/testing.go:2012 +0x5a4 fp=0x400007fe10 sp=0x400007fbe0 pc=0x7ff76d68b6a4
        runtime_test.TestMain(0x4000072820)
        	/Users/cherryyz/src/go/src/runtime/crash_test.go:29 +0x38 fp=0x400007fe80 sp=0x400007fe10 pc=0x7ff76d840398
        main.main()
        	_testmain.go:1525 +0x170 fp=0x400007ff40 sp=0x400007fe80 pc=0x7ff76d8e14f0
        runtime.main()
        	/Users/cherryyz/src/go/src/runtime/proc.go:271 +0x274 fp=0x400007ffd0 sp=0x400007ff40 pc=0x7ff76d5b15c4
        runtime.goexit({})
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400007ffd0 sp=0x400007ffd0 pc=0x7ff76d5eb444
        
        goroutine 2 gp=0x4000048680 m=nil [force gc (idle)]:
        runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        	/Users/cherryyz/src/go/src/runtime/proc.go:402 +0xc8 fp=0x400004bf90 sp=0x400004bf70 pc=0x7ff76d5b19f8
        runtime.goparkunlock(...)
        	/Users/cherryyz/src/go/src/runtime/proc.go:408
        runtime.forcegchelper()
        	/Users/cherryyz/src/go/src/runtime/proc.go:326 +0xb8 fp=0x400004bfd0 sp=0x400004bf90 pc=0x7ff76d5b1858
        runtime.goexit({})
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400004bfd0 sp=0x400004bfd0 pc=0x7ff76d5eb444
        created by runtime.init.6 in goroutine 1
        	/Users/cherryyz/src/go/src/runtime/proc.go:314 +0x24
        
        goroutine 3 gp=0x40000489c0 m=nil [GC sweep wait]:
        runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        	/Users/cherryyz/src/go/src/runtime/proc.go:402 +0xc8 fp=0x400004df60 sp=0x400004df40 pc=0x7ff76d5b19f8
        runtime.goparkunlock(...)
        	/Users/cherryyz/src/go/src/runtime/proc.go:408
        runtime.bgsweep(0x4000026070)
        	/Users/cherryyz/src/go/src/runtime/mgcsweep.go:277 +0xa0 fp=0x400004dfb0 sp=0x400004df60 pc=0x7ff76d599a40
        runtime.gcenable.gowrap1()
        	/Users/cherryyz/src/go/src/runtime/mgc.go:203 +0x28 fp=0x400004dfd0 sp=0x400004dfb0 pc=0x7ff76d58dba8
        runtime.goexit({})
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400004dfd0 sp=0x400004dfd0 pc=0x7ff76d5eb444
        created by runtime.gcenable in goroutine 1
        	/Users/cherryyz/src/go/src/runtime/mgc.go:203 +0x6c
        
        goroutine 4 gp=0x4000048b60 m=nil [GC scavenge wait]:
        runtime.gopark(0x4000026070?, 0x7ff76da3f948?, 0x1?, 0x0?, 0x4000048b60?)
        	/Users/cherryyz/src/go/src/runtime/proc.go:402 +0xc8 fp=0x400005ff60 sp=0x400005ff40 pc=0x7ff76d5b19f8
        runtime.goparkunlock(...)
        	/Users/cherryyz/src/go/src/runtime/proc.go:408
        runtime.(*scavengerState).park(0x7ff76dc99100)
        	/Users/cherryyz/src/go/src/runtime/mgcscavenge.go:425 +0x5c fp=0x400005ff90 sp=0x400005ff60 pc=0x7ff76d59744c
        runtime.bgscavenge(0x4000026070)
        	/Users/cherryyz/src/go/src/runtime/mgcscavenge.go:653 +0x44 fp=0x400005ffb0 sp=0x400005ff90 pc=0x7ff76d597994
        runtime.gcenable.gowrap2()
        	/Users/cherryyz/src/go/src/runtime/mgc.go:204 +0x28 fp=0x400005ffd0 sp=0x400005ffb0 pc=0x7ff76d58db48
        runtime.goexit({})
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400005ffd0 sp=0x400005ffd0 pc=0x7ff76d5eb444
        created by runtime.gcenable in goroutine 1
        	/Users/cherryyz/src/go/src/runtime/mgc.go:204 +0xac
        
        goroutine 5 gp=0x4000048ea0 m=nil [finalizer wait]:
        runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        	/Users/cherryyz/src/go/src/runtime/proc.go:402 +0xc8 fp=0x400004fd80 sp=0x400004fd60 pc=0x7ff76d5b19f8
        runtime.runfinq()
        	/Users/cherryyz/src/go/src/runtime/mfinal.go:193 +0x108 fp=0x400004ffd0 sp=0x400004fd80 pc=0x7ff76d58cc38
        runtime.goexit({})
        	/Users/cherryyz/src/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400004ffd0 sp=0x400004ffd0 pc=0x7ff76d5eb444
        created by runtime.createfing in goroutine 1
        	/Users/cherryyz/src/go/src/runtime/mfinal.go:163 +0x80
        r0   0x4000037ab0
        r1   0x3
        r2   0x4000037b10
        r3   0x23ccf8156a4
        r4   0x4
        r5   0x1
        r6   0x2
        r7   0x23ccf8066b0
        r8   0x418a073aa3bc2c75
        r9   0x418a073aa3bc2475
        r10  0x0
        r11  0x4
        r12  0x40000
        r13  0x0
        r14  0x0
        r15  0x10f
        r16  0x4000036000
        r17  0xd948203000
        r18  0x0
        r19  0x4000037b10
        r20  0x7ffce38a4670
        r21  0x7ffce33408b0
        r22  0x0
        r23  0x48
        r24  0x1
        r25  0x7ffce3300000
        r26  0x44
        r27  0x1
        r28  0x7ff76dc99520
        r29  0x4000037a60
        lr   0x7ffce33438b8
        sp   0x4000037a50
        pc   0x7ffce7a62740
        cpsr 0x20000000
--- FAIL: TestG0StackOverflow (4.52s)

@qmuntal
Copy link
Contributor

qmuntal commented Nov 13, 2023

@qmuntal do you have any insight to share?

In general, Windows doesn't like to handle exceptions that lie outside the system stack.

Other parts of the runtime already hit this limitation, for example, exceptions can only be resumed by functions within the system stack limits (workaround), and exceptions can't be unwinded if a function in the call stack is not within the system stack limits (workaround)

I couldn't find why we are seeing a EXCEPTION_STACK_OVERFLOW here, as I don't have access to my arm64 machine right now, but I supose that is just Windows being too cautios about the exception atack, as in the other examples I mentioned.

@cherrymui
Copy link
Member Author

@qmuntal Thanks! Do you know what's special about system allocated stacks? Is it possible to tell the system that some allocated memory is for system stack (maybe setting some flags on VirtualAlloc or something)? If not, do you think CL https://go.dev/cl/541997 (using some BSS space for stack) is a reasonable workaround? Alternatively maybe we could just say crash stack is not implemented on Windows/arm64, then for certain bad conditions (like morestack on g0 it will just print a message and exit without a stack trace, like before), unfortunate, but not terribly bad.
Thanks.

@qmuntal
Copy link
Contributor

qmuntal commented Nov 13, 2023

Thanks! Do you know what's special about system allocated stacks?

I don't know for sure, but my guess is that Windows is very strict about just using the system stack as a security hardening measure.

Is it possible to tell the system that some allocated memory is for system stack (maybe setting some flags on VirtualAlloc or something)?

Nop. What you can do is modify the thread stack limits stored in the TEB, which will trick Windows to think that some allocated memory is the system stack. But that's not documented, and I would try to avoid doing that.

If not, do you think CL https://go.dev/cl/541997 (using some BSS space for stack) is a reasonable workaround?

If the test passes, I don't have anything against CL 541997.

@qmuntal
Copy link
Contributor

qmuntal commented Nov 16, 2023

@dagood I recall seeing some similar failures in our windows/amd64 CI hosts. When you have time, can you check if CL 541997 also fixes them?

@dagood
Copy link
Contributor

dagood commented Nov 16, 2023

Yeah, I picked up on this too. I copied some logs to microsoft/go#1083. In that issue, the run includes retry logic that immediately runs the tests again on the same machine. The retries always seem to fail, which seems to point to this happening consistently on specific windows/amd64 machines and not on others. (Hitting the broader "Rerun failed jobs" button in Azure Pipelines randomly gets new agents, so this hasn't totally blocked anything yet.)

I wasn't able to find any agent difference in the logs, but it's very possible that the VMs running our pipelines are hosted on machines with varying specs or configurations without our knowledge.

Here's another run where I got rid of retries, where TestG0StackOverflow appears to be flaky on windows/amd64, and might knock on to cause other tests to fail: https://dev.azure.com/dnceng-public/public/_build/results?buildId=471869&view=logs&j=1aee57db-2891-55b4-47db-29f21c8a6fd5&t=97012fbd-1fec-5c8e-2c0d-3f28aaa85add&l=465


Unfortunately, If I add https://go.dev/cl/541997 as a patch and run that, it still times out and fails. It does print more info:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=471302&view=logs&j=14ce9276-1b71-5587-760e-edce85b6645f&t=8503eb12-65d8-5ec3-c57a-564f42e817be&l=321

Pasted log
2023-11-15T19:14:51.9821779Z --- FAIL: TestG0StackOverflow (928.64s)
2023-11-15T19:14:51.9823905Z     exec.go:193: test timed out while running command: C:\Users\CLOUDT~1\AppData\Local\Temp\go-build4130642704\b1434\runtime.test.exe -test.run=^TestG0StackOverflow$ -test.v
2023-11-15T19:14:51.9826411Z     crash_test.go:806: output:
2023-11-15T19:14:51.9826782Z         === RUN   TestG0StackOverflow
2023-11-15T19:14:51.9827468Z         runtime: morestack on g0, stack [0x45575fdd30 0x45575ffd60], sp=0x45575ff108, called from
2023-11-15T19:14:51.9828185Z         runtime.stackOverflow(0x94f0a665fedc?)
2023-11-15T19:14:51.9828999Z         	D:/a/_work/1/s/go/src/runtime/export_test.go:720 +0x4b fp=0x45575ff110 sp=0x45575ff108 pc=0xb6cd8b
2023-11-15T19:14:51.9829773Z         runtime.stackOverflow(0x2?)
2023-11-15T19:14:51.9830535Z         	D:/a/_work/1/s/go/src/runtime/export_test.go:722 +0x3d fp=0x45575ff228 sp=0x45575ff110 pc=0xb6cd7d
2023-11-15T19:14:51.9831421Z         runtime.stackOverflow(0x180637e0000?)
2023-11-15T19:14:51.9832209Z         	D:/a/_work/1/s/go/src/runtime/export_test.go:722 +0x3d fp=0x45575ff340 sp=0x45575ff228 pc=0xb6cd7d
2023-11-15T19:14:51.9833022Z         runtime.stackOverflow(0x1801e3d0000?)
2023-11-15T19:14:51.9833815Z         	D:/a/_work/1/s/go/src/runtime/export_test.go:722 +0x3d fp=0x45575ff458 sp=0x45575ff340 pc=0xb6cd7d
2023-11-15T19:14:51.9834592Z         runtime.stackOverflow(0x4e?)
2023-11-15T19:14:51.9835287Z         	D:/a/_work/1/s/go/src/runtime/export_test.go:722 +0x3d fp=0x45575ff570 sp=0x45575ff458 pc=0xb6cd7d
2023-11-15T19:14:51.9836146Z         runtime.stackOverflow(0x2080000?)
2023-11-15T19:14:51.9836893Z         	D:/a/_work/1/s/go/src/runtime/export_test.go:722 +0x3d fp=0x45575ff688 sp=0x45575ff570 pc=0xb6cd7d
2023-11-15T19:14:51.9837652Z         runtime.stackOverflow(0x0?)
2023-11-15T19:14:51.9838307Z         	D:/a/_work/1/s/go/src/runtime/export_test.go:722 +0x3d fp=0x45575ff7a0 sp=0x45575ff688 pc=0xb6cd7d
2023-11-15T19:14:51.9839207Z         runtime.stackOverflow(0x1801e3df9f0?)
2023-11-15T19:14:51.9839886Z         	D:/a/_work/1/s/go/src/runtime/export_test.go:722 +0x3d fp=0x45575ff8b8 sp=0x45575ff7a0 pc=0xb6cd7d
2023-11-15T19:14:51.9840656Z         runtime.stackOverflow(0xc1d3e5?)
2023-11-15T19:14:51.9841509Z         	D:/a/_work/1/s/go/src/runtime/export_test.go:722 +0x3d fp=0x45575ff9d0 sp=0x45575ff8b8 pc=0xb6cd7d
2023-11-15T19:14:51.9842377Z         runtime.stackOverflow(0x45575ffb18?)
2023-11-15T19:14:51.9843242Z         	D:/a/_work/1/s/go/src/runtime/export_test.go:722 +0x3d fp=0x45575ffae8 sp=0x45575ff9d0 pc=0xb6cd7d
2023-11-15T19:14:51.9844113Z         runtime.stackOverflow(0x0?)
2023-11-15T19:14:51.9844856Z         	D:/a/_work/1/s/go/src/runtime/export_test.go:722 +0x3d fp=0x45575ffc00 sp=0x45575ffae8 pc=0xb6cd7d
2023-11-15T19:14:51.9845686Z         runtime.stackOverflow(0xb7ae23?)
2023-11-15T19:14:51.9846507Z         	D:/a/_work/1/s/go/src/runtime/export_test.go:722 +0x3d fp=0x45575ffd18 sp=0x45575ffc00 pc=0xb6cd7d
2023-11-15T19:14:51.9847461Z         runtime_test.TestG0StackOverflow.G0StackOverflow.func1()
2023-11-15T19:14:51.9848293Z         	D:/a/_work/1/s/go/src/runtime/export_test.go:716 +0x36 fp=0x45575ffd30 sp=0x45575ffd18 pc=0xec0176
2023-11-15T19:14:51.9849097Z         runtime.systemstack(0x129db20)
2023-11-15T19:14:51.9849816Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:509 +0x49 fp=0x45575ffd40 sp=0x45575ffd30 pc=0xb791c9
2023-11-15T19:14:51.9850542Z         
2023-11-15T19:14:51.9850710Z         fatal error: morestack on g0
2023-11-15T19:14:51.9851022Z         
2023-11-15T19:14:51.9851214Z         runtime stack:
2023-11-15T19:14:51.9851480Z         runtime.throw({0xf884e4?, 0x0?})
2023-11-15T19:14:51.9852211Z         	D:/a/_work/1/s/go/src/runtime/panic.go:1016 +0x65 fp=0x13032c8 sp=0x1303298 pc=0xb3e625
2023-11-15T19:14:51.9852932Z         runtime.badmorestackg0.func1()
2023-11-15T19:14:51.9853593Z         	D:/a/_work/1/s/go/src/runtime/proc.go:531 +0xe5 fp=0x1303320 sp=0x13032c8 pc=0xb42965
2023-11-15T19:14:51.9854284Z         runtime.switchToCrashStack0()
2023-11-15T19:14:51.9854911Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:559 +0x37 fp=0x1303330 sp=0x1303320 pc=0xb79257
2023-11-15T19:14:51.9856177Z         
2023-11-15T19:14:51.9856373Z         goroutine 19 [running]:
2023-11-15T19:14:51.9856688Z         runtime.systemstack_switch()
2023-11-15T19:14:51.9857434Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc000061e60 sp=0xc000061e50 pc=0xb79168
2023-11-15T19:14:51.9858186Z         runtime.G0StackOverflow(...)
2023-11-15T19:14:51.9858581Z         	D:/a/_work/1/s/go/src/runtime/export_test.go:706
2023-11-15T19:14:51.9859053Z         runtime_test.TestG0StackOverflow(0xc000086820)
2023-11-15T19:14:51.9859822Z         	D:/a/_work/1/s/go/src/runtime/crash_test.go:828 +0x3bf fp=0xc000061f70 sp=0xc000061e60 pc=0xe1741f
2023-11-15T19:14:51.9860635Z         testing.tRunner(0xc000086820, 0xfb1ca8)
2023-11-15T19:14:51.9861403Z         	D:/a/_work/1/s/go/src/testing/testing.go:1682 +0xfb fp=0xc000061fc0 sp=0xc000061f70 pc=0xc1c47b
2023-11-15T19:14:51.9862166Z         testing.(*T).Run.gowrap1()
2023-11-15T19:14:51.9862913Z         	D:/a/_work/1/s/go/src/testing/testing.go:1735 +0x25 fp=0xc000061fe0 sp=0xc000061fc0 pc=0xc1d3e5
2023-11-15T19:14:51.9863689Z         runtime.goexit({})
2023-11-15T19:14:51.9864323Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:1694 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0xb7b1c1
2023-11-15T19:14:51.9865097Z         created by testing.(*T).Run in goroutine 1
2023-11-15T19:14:51.9865533Z         	D:/a/_work/1/s/go/src/testing/testing.go:1735 +0x390
2023-11-15T19:14:51.9865943Z         
2023-11-15T19:14:51.9866139Z         goroutine 1 [chan receive]:
2023-11-15T19:14:51.9866594Z         runtime.gopark(0xc0000c7980?, 0xb10a17?, 0x80?, 0x41?, 0x18?)
2023-11-15T19:14:51.9867392Z         	D:/a/_work/1/s/go/src/runtime/proc.go:400 +0xce fp=0xc0000c7918 sp=0xc0000c78f8 pc=0xb4210e
2023-11-15T19:14:51.9868349Z         runtime.chanrecv(0xc00010c150, 0xc0000c79ff, 0x1)
2023-11-15T19:14:51.9869125Z         	D:/a/_work/1/s/go/src/runtime/chan.go:583 +0x3cd fp=0xc0000c7990 sp=0xc0000c7918 pc=0xb08eed
2023-11-15T19:14:51.9869873Z         runtime.chanrecv1(0x129d0c0?, 0xef52a0?)
2023-11-15T19:14:51.9870567Z         	D:/a/_work/1/s/go/src/runtime/chan.go:442 +0x12 fp=0xc0000c79b8 sp=0xc0000c7990 pc=0xb08af2
2023-11-15T19:14:51.9871449Z         testing.(*T).Run(0xc000086680, {0xf8b95f?, 0xc03275e284?}, 0xfb1ca8)
2023-11-15T19:14:51.9872369Z         	D:/a/_work/1/s/go/src/testing/testing.go:1736 +0x3ab fp=0xc0000c7a78 sp=0xc0000c79b8 pc=0xc1d28b
2023-11-15T19:14:51.9873272Z         testing.runTests.func1(0xc000086680)
2023-11-15T19:14:51.9873988Z         	D:/a/_work/1/s/go/src/testing/testing.go:2146 +0x37 fp=0xc0000c7ab8 sp=0xc0000c7a78 pc=0xc1f417
2023-11-15T19:14:51.9874973Z         testing.tRunner(0xc000086680, 0xc0000c7bd0)
2023-11-15T19:14:51.9875748Z         	D:/a/_work/1/s/go/src/testing/testing.go:1682 +0xfb fp=0xc0000c7b08 sp=0xc0000c7ab8 pc=0xc1c47b
2023-11-15T19:14:51.9876793Z         testing.runTests(0xc0000a4120, {0x12992a0?, 0x1d8, 0x1d8}, {0xb18fc8?, 0xc0000d5078?, 0x0?})
2023-11-15T19:14:51.9877858Z         	D:/a/_work/1/s/go/src/testing/testing.go:2144 +0x456 fp=0xc0000c7c00 sp=0xc0000c7b08 pc=0xc1f2f6
2023-11-15T19:14:51.9878671Z         testing.(*M).Run(0xc0000a08c0)
2023-11-15T19:14:51.9879401Z         	D:/a/_work/1/s/go/src/testing/testing.go:2012 +0x68b fp=0xc0000c7e30 sp=0xc0000c7c00 pc=0xc1dccb
2023-11-15T19:14:51.9880174Z         runtime_test.TestMain(0xc0000a08c0)
2023-11-15T19:14:51.9880918Z         	D:/a/_work/1/s/go/src/runtime/crash_test.go:29 +0x37 fp=0xc0000c7e98 sp=0xc0000c7e30 pc=0xe11537
2023-11-15T19:14:51.9881624Z         main.main()
2023-11-15T19:14:51.9882039Z         	_testmain.go:1621 +0x195 fp=0xc0000c7f50 sp=0xc0000c7e98 pc=0xec6f55
2023-11-15T19:14:51.9882583Z         runtime.main()
2023-11-15T19:14:51.9883183Z         	D:/a/_work/1/s/go/src/runtime/proc.go:269 +0x28b fp=0xc0000c7fe0 sp=0xc0000c7f50 pc=0xb41ceb
2023-11-15T19:14:51.9883936Z         runtime.goexit({})
2023-11-15T19:14:51.9884590Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:1694 +0x1 fp=0xc0000c7fe8 sp=0xc0000c7fe0 pc=0xb7b1c1
2023-11-15T19:14:51.9885413Z         
2023-11-15T19:14:51.9885614Z         goroutine 2 [force gc (idle)]:
2023-11-15T19:14:51.9885964Z         runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
2023-11-15T19:14:51.9886697Z         	D:/a/_work/1/s/go/src/runtime/proc.go:400 +0xce fp=0xc00004ffa8 sp=0xc00004ff88 pc=0xb4210e
2023-11-15T19:14:51.9887414Z         runtime.goparkunlock(...)
2023-11-15T19:14:51.9887769Z         	D:/a/_work/1/s/go/src/runtime/proc.go:406
2023-11-15T19:14:51.9888127Z         runtime.forcegchelper()
2023-11-15T19:14:51.9888864Z         	D:/a/_work/1/s/go/src/runtime/proc.go:324 +0xb8 fp=0xc00004ffe0 sp=0xc00004ffa8 pc=0xb41f78
2023-11-15T19:14:51.9889592Z         runtime.goexit({})
2023-11-15T19:14:51.9890227Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:1694 +0x1 fp=0xc00004ffe8 sp=0xc00004ffe0 pc=0xb7b1c1
2023-11-15T19:14:51.9890962Z         created by runtime.init.6 in goroutine 1
2023-11-15T19:14:51.9891402Z         	D:/a/_work/1/s/go/src/runtime/proc.go:312 +0x1a
2023-11-15T19:14:51.9891805Z         
2023-11-15T19:14:51.9891968Z         goroutine 3 [GC sweep wait]:
2023-11-15T19:14:51.9892343Z         runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
2023-11-15T19:14:51.9893077Z         	D:/a/_work/1/s/go/src/runtime/proc.go:400 +0xce fp=0xc000051f80 sp=0xc000051f60 pc=0xb4210e
2023-11-15T19:14:51.9893793Z         runtime.goparkunlock(...)
2023-11-15T19:14:51.9894190Z         	D:/a/_work/1/s/go/src/runtime/proc.go:406
2023-11-15T19:14:51.9894612Z         runtime.bgsweep(0xc00005e000)
2023-11-15T19:14:51.9895337Z         	D:/a/_work/1/s/go/src/runtime/mgcsweep.go:276 +0x94 fp=0xc000051fc8 sp=0xc000051f80 pc=0xb29d94
2023-11-15T19:14:51.9896192Z         runtime.gcenable.gowrap1()
2023-11-15T19:14:51.9896811Z         	D:/a/_work/1/s/go/src/runtime/mgc.go:203 +0x25 fp=0xc000051fe0 sp=0xc000051fc8 pc=0xb1e8c5
2023-11-15T19:14:51.9897521Z         runtime.goexit({})
2023-11-15T19:14:51.9898205Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:1694 +0x1 fp=0xc000051fe8 sp=0xc000051fe0 pc=0xb7b1c1
2023-11-15T19:14:51.9899022Z         created by runtime.gcenable in goroutine 1
2023-11-15T19:14:51.9899518Z         	D:/a/_work/1/s/go/src/runtime/mgc.go:203 +0x66
2023-11-15T19:14:51.9899913Z         
2023-11-15T19:14:51.9900113Z         goroutine 4 [GC scavenge wait]:
2023-11-15T19:14:51.9900630Z         runtime.gopark(0xc00005e000?, 0x102ae80?, 0x1?, 0x0?, 0xc00004cb60?)
2023-11-15T19:14:51.9901477Z         	D:/a/_work/1/s/go/src/runtime/proc.go:400 +0xce fp=0xc000065f78 sp=0xc000065f58 pc=0xb4210e
2023-11-15T19:14:51.9902196Z         runtime.goparkunlock(...)
2023-11-15T19:14:51.9902544Z         	D:/a/_work/1/s/go/src/runtime/proc.go:406
2023-11-15T19:14:51.9902983Z         runtime.(*scavengerState).park(0x129d8a0)
2023-11-15T19:14:51.9903755Z         	D:/a/_work/1/s/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000065fa8 sp=0xc000065f78 pc=0xb27789
2023-11-15T19:14:51.9904596Z         runtime.bgscavenge(0xc00005e000)
2023-11-15T19:14:51.9905415Z         	D:/a/_work/1/s/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000065fc8 sp=0xc000065fa8 pc=0xb27d1c
2023-11-15T19:14:51.9906205Z         runtime.gcenable.gowrap2()
2023-11-15T19:14:51.9906824Z         	D:/a/_work/1/s/go/src/runtime/mgc.go:204 +0x25 fp=0xc000065fe0 sp=0xc000065fc8 pc=0xb1e865
2023-11-15T19:14:51.9907527Z         runtime.goexit({})
2023-11-15T19:14:51.9908155Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:1694 +0x1 fp=0xc000065fe8 sp=0xc000065fe0 pc=0xb7b1c1
2023-11-15T19:14:51.9908922Z         created by runtime.gcenable in goroutine 1
2023-11-15T19:14:51.9909333Z         	D:/a/_work/1/s/go/src/runtime/mgc.go:204 +0xa5
2023-11-15T19:14:51.9909718Z         
2023-11-15T19:14:51.9909994Z         goroutine 18 [finalizer wait]:
2023-11-15T19:14:51.9910499Z         runtime.gopark(0xc000053e60?, 0xb26a25?, 0xa0?, 0xb5?, 0x400000?)
2023-11-15T19:14:51.9911338Z         	D:/a/_work/1/s/go/src/runtime/proc.go:400 +0xce fp=0xc000053e20 sp=0xc000053e00 pc=0xb4210e
2023-11-15T19:14:51.9912056Z         runtime.runfinq()
2023-11-15T19:14:51.9912777Z         	D:/a/_work/1/s/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000053fe0 sp=0xc000053e20 pc=0xb1d8e7
2023-11-15T19:14:51.9913523Z         runtime.goexit({})
2023-11-15T19:14:51.9914126Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:1694 +0x1 fp=0xc000053fe8 sp=0xc000053fe0 pc=0xb7b1c1
2023-11-15T19:14:51.9914912Z         created by runtime.createfing in goroutine 1
2023-11-15T19:14:51.9915456Z         	D:/a/_work/1/s/go/src/runtime/mfinal.go:163 +0x3d
2023-11-15T19:14:51.9915962Z         Exception 0xc00000fd 0x0 0x4557403000 0x7ffdb6e25047
2023-11-15T19:14:51.9916361Z         PC=0x7ffdb6e25047
2023-11-15T19:14:51.9916617Z         
2023-11-15T19:14:51.9916843Z         runtime: g 0: unknown pc 0x7ffdb6e25047
2023-11-15T19:14:51.9917339Z         stack: frame={sp:0x1302ae8, fp:0x0} stack=[0x0,0x1303340)
2023-11-15T19:14:51.9917903Z         0x00000000013029e8:  0x0000000000000000  0x000001801e3d0001 
2023-11-15T19:14:51.9918521Z         0x00000000013029f8:  0x00007393551bd466  0x0000000000000004 
2023-11-15T19:14:51.9919133Z         0x0000000001302a08:  0x0000000001302b00  0x000001801e3e858a 
2023-11-15T19:14:51.9919743Z         0x0000000001302a18:  0x000000000000000a  0x000001801e3e1378 
2023-11-15T19:14:51.9920374Z         0x0000000001302a28:  0x00007ffdb458949d  0x0000000000000080 
2023-11-15T19:14:51.9921006Z         0x0000000001302a38:  0x00007ffdb6de90c9  0x000001801e3e1378 
2023-11-15T19:14:51.9921626Z         0x0000000001302a48:  0x0000000001302aa9  0x0000000000000001 
2023-11-15T19:14:51.9922238Z         0x0000000001302a58:  0x0000000000000000  0x000001801e3e8594 
2023-11-15T19:14:51.9922817Z         0x0000000001302a68:  0x00007ffd00000001  0x0000000000000004 
2023-11-15T19:14:51.9923529Z         0x0000000001302a78:  0x00000001b4010004  0x0000000000000004 
2023-11-15T19:14:51.9924146Z         0x0000000001302a88:  0x0000000000000000  0x0000000001302bd8 
2023-11-15T19:14:51.9924766Z         0x0000000001302a98:  0x00000001b4000004  0x000001801e3e0004 
2023-11-15T19:14:51.9925354Z         0x0000000001302aa8:  0x0000000000000000  0x0000000001302bd0 
2023-11-15T19:14:51.9926047Z         0x0000000001302ab8:  0x00007393551bd786  0x0000000001302bd0 
2023-11-15T19:14:51.9926673Z         0x0000000001302ac8:  0x00007ffdb4587a89  0x000000000129eb60 
2023-11-15T19:14:51.9927288Z         0x0000000001302ad8:  0x00007ffdb44eafc0  0x0000000000000004 
2023-11-15T19:14:51.9927874Z         0x0000000001302ae8: <0x0000000001302c50  0x0000000001302bd0 
2023-11-15T19:14:51.9928489Z         0x0000000001302af8:  0x00007ffdb45894de  0x0000000000000001 
2023-11-15T19:14:51.9929091Z         0x0000000001302b08:  0x00007ffdb44d0000  0x000001801e3e1370 
2023-11-15T19:14:51.9929700Z         0x0000000001302b18:  0x0000000000000000  0x0000000001302b90 
2023-11-15T19:14:51.9930272Z         0x0000000001302b28:  0x00007ffdb4588dba  0x000001801e3e1378 
2023-11-15T19:14:51.9930909Z         0x0000000001302b38:  0x0000000001302bd0  0x0000000000000000 
2023-11-15T19:14:51.9931272Z         0x0000000001302b48:  0x0000000000000000  0x418a073aa3bc1c75 
2023-11-15T19:14:51.9931623Z         0x0000000001302b58:  0x418a073aa3bc2475  0x418a073aa3bc2c75 
2023-11-15T19:14:51.9931945Z         0x0000000001302b68:  0x00007ffdb4561043  0x00007ffdb4589f00 
2023-11-15T19:14:51.9932294Z         0x0000000001302b78:  0x000001801e3e1370  0x00007393551bd686 
2023-11-15T19:14:51.9932640Z         0x0000000001302b88:  0x0000000000000001  0x0000000000000000 
2023-11-15T19:14:51.9932986Z         0x0000000001302b98:  0x00007ffdb458916a  0x000001801e3e1370 
2023-11-15T19:14:51.9933308Z         0x0000000001302ba8:  0x000001801e3e1370  0x0000000000000000 
2023-11-15T19:14:51.9933662Z         0x0000000001302bb8:  0x0000000000000005  0x000001801e3e1370 
2023-11-15T19:14:51.9934011Z         0x0000000001302bc8:  0x00007ffdb4561043  0x0004000100040000 
2023-11-15T19:14:51.9934357Z         0x0000000001302bd8:  0x0000000000000000  0x0000000000000004 
2023-11-15T19:14:51.9934631Z         runtime: g 0: unknown pc 0x7ffdb6e25047
2023-11-15T19:14:51.9934988Z         stack: frame={sp:0x1302ae8, fp:0x0} stack=[0x0,0x1303340)
2023-11-15T19:14:51.9935337Z         0x00000000013029e8:  0x0000000000000000  0x000001801e3d0001 
2023-11-15T19:14:51.9935688Z         0x00000000013029f8:  0x00007393551bd466  0x0000000000000004 
2023-11-15T19:14:51.9936237Z         0x0000000001302a08:  0x0000000001302b00  0x000001801e3e858a 
2023-11-15T19:14:51.9936594Z         0x0000000001302a18:  0x000000000000000a  0x000001801e3e1378 
2023-11-15T19:14:51.9936945Z         0x0000000001302a28:  0x00007ffdb458949d  0x0000000000000080 
2023-11-15T19:14:51.9937297Z         0x0000000001302a38:  0x00007ffdb6de90c9  0x000001801e3e1378 
2023-11-15T19:14:51.9937621Z         0x0000000001302a48:  0x0000000001302aa9  0x0000000000000001 
2023-11-15T19:14:51.9937967Z         0x0000000001302a58:  0x0000000000000000  0x000001801e3e8594 
2023-11-15T19:14:51.9938320Z         0x0000000001302a68:  0x00007ffd00000001  0x0000000000000004 
2023-11-15T19:14:51.9938681Z         0x0000000001302a78:  0x00000001b4010004  0x0000000000000004 
2023-11-15T19:14:51.9939001Z         0x0000000001302a88:  0x0000000000000000  0x0000000001302bd8 
2023-11-15T19:14:51.9939353Z         0x0000000001302a98:  0x00000001b4000004  0x000001801e3e0004 
2023-11-15T19:14:51.9939703Z         0x0000000001302aa8:  0x0000000000000000  0x0000000001302bd0 
2023-11-15T19:14:51.9940048Z         0x0000000001302ab8:  0x00007393551bd786  0x0000000001302bd0 
2023-11-15T19:14:51.9940369Z         0x0000000001302ac8:  0x00007ffdb4587a89  0x000000000129eb60 
2023-11-15T19:14:51.9940716Z         0x0000000001302ad8:  0x00007ffdb44eafc0  0x0000000000000004 
2023-11-15T19:14:51.9941208Z         0x0000000001302ae8: <0x0000000001302c50  0x0000000001302bd0 
2023-11-15T19:14:51.9941696Z         0x0000000001302af8:  0x00007ffdb45894de  0x0000000000000001 
2023-11-15T19:14:51.9942021Z         0x0000000001302b08:  0x00007ffdb44d0000  0x000001801e3e1370 
2023-11-15T19:14:51.9942370Z         0x0000000001302b18:  0x0000000000000000  0x0000000001302b90 
2023-11-15T19:14:51.9942721Z         0x0000000001302b28:  0x00007ffdb4588dba  0x000001801e3e1378 
2023-11-15T19:14:51.9943080Z         0x0000000001302b38:  0x0000000001302bd0  0x0000000000000000 
2023-11-15T19:14:51.9943403Z         0x0000000001302b48:  0x0000000000000000  0x418a073aa3bc1c75 
2023-11-15T19:14:51.9943752Z         0x0000000001302b58:  0x418a073aa3bc2475  0x418a073aa3bc2c75 
2023-11-15T19:14:51.9944105Z         0x0000000001302b68:  0x00007ffdb4561043  0x00007ffdb4589f00 
2023-11-15T19:14:51.9944454Z         0x0000000001302b78:  0x000001801e3e1370  0x00007393551bd686 
2023-11-15T19:14:51.9944803Z         0x0000000001302b88:  0x0000000000000001  0x0000000000000000 
2023-11-15T19:14:51.9945124Z         0x0000000001302b98:  0x00007ffdb458916a  0x000001801e3e1370 
2023-11-15T19:14:51.9945474Z         0x0000000001302ba8:  0x000001801e3e1370  0x0000000000000000 
2023-11-15T19:14:51.9945824Z         0x0000000001302bb8:  0x0000000000000005  0x000001801e3e1370 
2023-11-15T19:14:51.9946171Z         0x0000000001302bc8:  0x00007ffdb4561043  0x0004000100040000 
2023-11-15T19:14:51.9946730Z         0x0000000001302bd8:  0x0000000000000000  0x0000000000000004 
2023-11-15T19:14:51.9947014Z         
2023-11-15T19:14:51.9947155Z         goroutine 19 [running]:
2023-11-15T19:14:51.9947315Z         runtime.systemstack_switch()
2023-11-15T19:14:51.9947720Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc000061e60 sp=0xc000061e50 pc=0xb79168
2023-11-15T19:14:51.9948143Z         runtime.G0StackOverflow(...)
2023-11-15T19:14:51.9948379Z         	D:/a/_work/1/s/go/src/runtime/export_test.go:706
2023-11-15T19:14:51.9948634Z         runtime_test.TestG0StackOverflow(0xc000086820)
2023-11-15T19:14:51.9949099Z         	D:/a/_work/1/s/go/src/runtime/crash_test.go:828 +0x3bf fp=0xc000061f70 sp=0xc000061e60 pc=0xe1741f
2023-11-15T19:14:51.9949553Z         testing.tRunner(0xc000086820, 0xfb1ca8)
2023-11-15T19:14:51.9949984Z         	D:/a/_work/1/s/go/src/testing/testing.go:1682 +0xfb fp=0xc000061fc0 sp=0xc000061f70 pc=0xc1c47b
2023-11-15T19:14:51.9950460Z         testing.(*T).Run.gowrap1()
2023-11-15T19:14:51.9950861Z         	D:/a/_work/1/s/go/src/testing/testing.go:1735 +0x25 fp=0xc000061fe0 sp=0xc000061fc0 pc=0xc1d3e5
2023-11-15T19:14:51.9951287Z         runtime.goexit({})
2023-11-15T19:14:51.9951812Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:1694 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0xb7b1c1
2023-11-15T19:14:51.9952273Z         created by testing.(*T).Run in goroutine 1
2023-11-15T19:14:51.9952521Z         	D:/a/_work/1/s/go/src/testing/testing.go:1735 +0x390
2023-11-15T19:14:51.9952768Z         
2023-11-15T19:14:51.9952892Z         goroutine 1 [chan receive]:
2023-11-15T19:14:51.9953130Z         runtime.gopark(0xc0000c7980?, 0xb10a17?, 0x80?, 0x41?, 0x18?)
2023-11-15T19:14:51.9953608Z         	D:/a/_work/1/s/go/src/runtime/proc.go:400 +0xce fp=0xc0000c7918 sp=0xc0000c78f8 pc=0xb4210e
2023-11-15T19:14:51.9954056Z         runtime.chanrecv(0xc00010c150, 0xc0000c79ff, 0x1)
2023-11-15T19:14:51.9954504Z         	D:/a/_work/1/s/go/src/runtime/chan.go:583 +0x3cd fp=0xc0000c7990 sp=0xc0000c7918 pc=0xb08eed
2023-11-15T19:14:51.9954909Z         runtime.chanrecv1(0x129d0c0?, 0xef52a0?)
2023-11-15T19:14:51.9955325Z         	D:/a/_work/1/s/go/src/runtime/chan.go:442 +0x12 fp=0xc0000c79b8 sp=0xc0000c7990 pc=0xb08af2
2023-11-15T19:14:51.9955821Z         testing.(*T).Run(0xc000086680, {0xf8b95f?, 0xc03275e284?}, 0xfb1ca8)
2023-11-15T19:14:51.9956341Z         	D:/a/_work/1/s/go/src/testing/testing.go:1736 +0x3ab fp=0xc0000c7a78 sp=0xc0000c79b8 pc=0xc1d28b
2023-11-15T19:14:51.9956875Z         testing.runTests.func1(0xc000086680)
2023-11-15T19:14:51.9957414Z         	D:/a/_work/1/s/go/src/testing/testing.go:2146 +0x37 fp=0xc0000c7ab8 sp=0xc0000c7a78 pc=0xc1f417
2023-11-15T19:14:51.9957883Z         testing.tRunner(0xc000086680, 0xc0000c7bd0)
2023-11-15T19:14:51.9958327Z         	D:/a/_work/1/s/go/src/testing/testing.go:1682 +0xfb fp=0xc0000c7b08 sp=0xc0000c7ab8 pc=0xc1c47b
2023-11-15T19:14:51.9958933Z         testing.runTests(0xc0000a4120, {0x12992a0?, 0x1d8, 0x1d8}, {0xb18fc8?, 0xc0000d5078?, 0x0?})
2023-11-15T19:14:51.9959511Z         	D:/a/_work/1/s/go/src/testing/testing.go:2144 +0x456 fp=0xc0000c7c00 sp=0xc0000c7b08 pc=0xc1f2f6
2023-11-15T19:14:51.9959953Z         testing.(*M).Run(0xc0000a08c0)
2023-11-15T19:14:51.9960367Z         	D:/a/_work/1/s/go/src/testing/testing.go:2012 +0x68b fp=0xc0000c7e30 sp=0xc0000c7c00 pc=0xc1dccb
2023-11-15T19:14:51.9960808Z         runtime_test.TestMain(0xc0000a08c0)
2023-11-15T19:14:51.9961213Z         	D:/a/_work/1/s/go/src/runtime/crash_test.go:29 +0x37 fp=0xc0000c7e98 sp=0xc0000c7e30 pc=0xe11537
2023-11-15T19:14:51.9961645Z         main.main()
2023-11-15T19:14:51.9961891Z         	_testmain.go:1621 +0x195 fp=0xc0000c7f50 sp=0xc0000c7e98 pc=0xec6f55
2023-11-15T19:14:51.9962376Z         runtime.main()
2023-11-15T19:14:51.9962700Z         	D:/a/_work/1/s/go/src/runtime/proc.go:269 +0x28b fp=0xc0000c7fe0 sp=0xc0000c7f50 pc=0xb41ceb
2023-11-15T19:14:51.9963120Z         runtime.goexit({})
2023-11-15T19:14:51.9963488Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:1694 +0x1 fp=0xc0000c7fe8 sp=0xc0000c7fe0 pc=0xb7b1c1
2023-11-15T19:14:51.9963901Z         
2023-11-15T19:14:51.9963999Z         goroutine 2 [force gc (idle)]:
2023-11-15T19:14:51.9964227Z         runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
2023-11-15T19:14:51.9964651Z         	D:/a/_work/1/s/go/src/runtime/proc.go:400 +0xce fp=0xc00004ffa8 sp=0xc00004ff88 pc=0xb4210e
2023-11-15T19:14:51.9965066Z         runtime.goparkunlock(...)
2023-11-15T19:14:51.9965247Z         	D:/a/_work/1/s/go/src/runtime/proc.go:406
2023-11-15T19:14:51.9965518Z         runtime.forcegchelper()
2023-11-15T19:14:51.9966203Z         	D:/a/_work/1/s/go/src/runtime/proc.go:324 +0xb8 fp=0xc00004ffe0 sp=0xc00004ffa8 pc=0xb41f78
2023-11-15T19:14:51.9966610Z         runtime.goexit({})
2023-11-15T19:14:51.9966949Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:1694 +0x1 fp=0xc00004ffe8 sp=0xc00004ffe0 pc=0xb7b1c1
2023-11-15T19:14:51.9967619Z         created by runtime.init.6 in goroutine 1
2023-11-15T19:14:51.9967890Z         	D:/a/_work/1/s/go/src/runtime/proc.go:312 +0x1a
2023-11-15T19:14:51.9968128Z         
2023-11-15T19:14:51.9968223Z         goroutine 3 [GC sweep wait]:
2023-11-15T19:14:51.9968445Z         runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
2023-11-15T19:14:51.9968867Z         	D:/a/_work/1/s/go/src/runtime/proc.go:400 +0xce fp=0xc000051f80 sp=0xc000051f60 pc=0xb4210e
2023-11-15T19:14:51.9969278Z         runtime.goparkunlock(...)
2023-11-15T19:14:51.9969458Z         	D:/a/_work/1/s/go/src/runtime/proc.go:406
2023-11-15T19:14:51.9969700Z         runtime.bgsweep(0xc00005e000)
2023-11-15T19:14:51.9970107Z         	D:/a/_work/1/s/go/src/runtime/mgcsweep.go:276 +0x94 fp=0xc000051fc8 sp=0xc000051f80 pc=0xb29d94
2023-11-15T19:14:51.9970537Z         runtime.gcenable.gowrap1()
2023-11-15T19:14:51.9970887Z         	D:/a/_work/1/s/go/src/runtime/mgc.go:203 +0x25 fp=0xc000051fe0 sp=0xc000051fc8 pc=0xb1e8c5
2023-11-15T19:14:51.9971293Z         runtime.goexit({})
2023-11-15T19:14:51.9971660Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:1694 +0x1 fp=0xc000051fe8 sp=0xc000051fe0 pc=0xb7b1c1
2023-11-15T19:14:51.9972102Z         created by runtime.gcenable in goroutine 1
2023-11-15T19:14:51.9972336Z         	D:/a/_work/1/s/go/src/runtime/mgc.go:203 +0x66
2023-11-15T19:14:51.9972590Z         
2023-11-15T19:14:51.9972799Z         goroutine 4 [GC scavenge wait]:
2023-11-15T19:14:51.9973122Z         runtime.gopark(0xc00005e000?, 0x102ae80?, 0x1?, 0x0?, 0xc00004cb60?)
2023-11-15T19:14:51.9973629Z         	D:/a/_work/1/s/go/src/runtime/proc.go:400 +0xce fp=0xc000065f78 sp=0xc000065f58 pc=0xb4210e
2023-11-15T19:14:51.9974090Z         runtime.goparkunlock(...)
2023-11-15T19:14:51.9974298Z         	D:/a/_work/1/s/go/src/runtime/proc.go:406
2023-11-15T19:14:51.9974523Z         runtime.(*scavengerState).park(0x129d8a0)
2023-11-15T19:14:51.9974985Z         	D:/a/_work/1/s/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000065fa8 sp=0xc000065f78 pc=0xb27789
2023-11-15T19:14:51.9975445Z         runtime.bgscavenge(0xc00005e000)
2023-11-15T19:14:51.9975882Z         	D:/a/_work/1/s/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000065fc8 sp=0xc000065fa8 pc=0xb27d1c
2023-11-15T19:14:51.9976328Z         runtime.gcenable.gowrap2()
2023-11-15T19:14:51.9976678Z         	D:/a/_work/1/s/go/src/runtime/mgc.go:204 +0x25 fp=0xc000065fe0 sp=0xc000065fc8 pc=0xb1e865
2023-11-15T19:14:51.9977078Z         runtime.goexit({})
2023-11-15T19:14:51.9977440Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:1694 +0x1 fp=0xc000065fe8 sp=0xc000065fe0 pc=0xb7b1c1
2023-11-15T19:14:51.9977959Z         created by runtime.gcenable in goroutine 1
2023-11-15T19:14:51.9978274Z         	D:/a/_work/1/s/go/src/runtime/mgc.go:204 +0xa5
2023-11-15T19:14:51.9978504Z         
2023-11-15T19:14:51.9978630Z         goroutine 18 [finalizer wait]:
2023-11-15T19:14:51.9978888Z         runtime.gopark(0xc000053e60?, 0xb26a25?, 0xa0?, 0xb5?, 0x400000?)
2023-11-15T19:14:51.9979386Z         	D:/a/_work/1/s/go/src/runtime/proc.go:400 +0xce fp=0xc000053e20 sp=0xc000053e00 pc=0xb4210e
2023-11-15T19:14:51.9979794Z         runtime.runfinq()
2023-11-15T19:14:51.9980163Z         	D:/a/_work/1/s/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000053fe0 sp=0xc000053e20 pc=0xb1d8e7
2023-11-15T19:14:51.9980583Z         runtime.goexit({})
2023-11-15T19:14:51.9980930Z         	D:/a/_work/1/s/go/src/runtime/asm_amd64.s:1694 +0x1 fp=0xc000053fe8 sp=0xc000053fe0 pc=0xb7b1c1
2023-11-15T19:14:51.9981402Z         created by runtime.createfing in goroutine 1
2023-11-15T19:14:51.9981682Z         	D:/a/_work/1/s/go/src/runtime/mfinal.go:163 +0x3d
2023-11-15T19:14:51.9981898Z         rax     0x1140
2023-11-15T19:14:51.9982043Z         rbx     0x1302bd0
2023-11-15T19:14:51.9982196Z         rcx     0x1302b50
2023-11-15T19:14:51.9982348Z         rdx     0x3
2023-11-15T19:14:51.9982458Z         rdi     0x1801e3e1370
2023-11-15T19:14:51.9982687Z         rsi     0x0
2023-11-15T19:14:51.9982821Z         rbp     0x1301ac0
2023-11-15T19:14:51.9982976Z         rsp     0x1302ae8
2023-11-15T19:14:51.9983121Z         r8      0x1302bd0
2023-11-15T19:14:51.9983408Z         r9      0x1
2023-11-15T19:14:51.9983547Z         r10     0x1301000
2023-11-15T19:14:51.9983676Z         r11     0x4557403000
2023-11-15T19:14:51.9983844Z         r12     0x7ffdb44eafc0
2023-11-15T19:14:51.9984011Z         r13     0x129eb60
2023-11-15T19:14:51.9984165Z         r14     0x7ffdb44d0000
2023-11-15T19:14:51.9984307Z         r15     0x1
2023-11-15T19:14:51.9984449Z         rip     0x7ffdb6e25047
2023-11-15T19:14:51.9984622Z         rflags  0x10287
2023-11-15T19:14:51.9984741Z         cs      0x33
2023-11-15T19:14:51.9984879Z         fs      0x53
2023-11-15T19:14:51.9985011Z         gs      0x2b
2023-11-15T19:14:51.9985146Z FAIL
2023-11-15T19:14:51.9985233Z FAIL	runtime	1026.357s

In this run, all windows testing jobs failed, which could be up to chance, but seems unusual.

I've kicked off a few more runs with the patch to get more data:

@cherrymui
Copy link
Member Author

Thanks for testing. The failure is similar to the one above #63938 (comment) , Exception 0xc00000fd EXCEPTION_STACK_OVERFLOW. So I guess even static allocation doesn't always make Windows happy.

One possibility may be that if we're crashing on a non-system allocated crash stack, after dumping stack traces, don't throw an exception, just exit. Maybe this will cause a different exit code, but maybe that is fine? Another possibility is disabling crash stack on Windows, so it works as before, just print a message (without stack trace) and abort.

@qmuntal @dagood do you know why EXCEPTION_STACK_OVERFLOW will cause the program hang? If it just throws an EXCEPTION_STACK_OVERFLOW but not hang, I'd be okay with that. Thanks.

@qmuntal
Copy link
Contributor

qmuntal commented Nov 19, 2023

If it just throws an EXCEPTION_STACK_OVERFLOW but not hang, I'd be okay with that. Thanks.

Could you try calling SetThreadStackGuarantee with the expected signal stack size? That might help avoiding the hand on the stack overflow exception handler.

If this doesn't help and no one have another solution, then I propose to revert this functionality (on Windows) and investigate it more for go 1.23, as I'm pretty sure it can be done.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/543996 mentions this issue: runtime: disable crash stack on Windows

@cherrymui
Copy link
Member Author

Thanks. Sent CL https://go.dev/cl/543996 to switch back to the old behavior on Windows for now. We'll try again in Go 1.23.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch-arm64 compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows release-blocker
Projects
None yet
Development

No branches or pull requests

6 participants