Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: occasional hangs in TestCgoCallbackGC #32023

Open
bcmills opened this issue May 14, 2019 · 18 comments
Open

runtime: occasional hangs in TestCgoCallbackGC #32023

bcmills opened this issue May 14, 2019 · 18 comments
Labels
NeedsInvestigation Testing
Milestone
Backlog

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented May 14, 2019

Looks like the test hung for some reason.

https://build.golang.org/log/e9e9a00d8fbae5c7ffb803d871b86982849c08ea

--- FAIL: TestCgoCallbackGC (60.03s)
    crash_test.go:95: testprogcgo CgoCallbackGC exit status: exit status 2
    crash_cgo_test.go:70: expected "OK\n", but got:
        SIGQUIT: quit
        PC=0x1005d49c m=0 sigcode=0
        
        goroutine 0 [idle]:
        runtime.futex(0x101f8a08, 0x80, 0x0, 0x0, 0x0, 0x0, 0x10033d90, 0x10035fc8, 0x10033de8, 0x100000081, ...)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/sys_linux_ppc64x.s:472 +0x1c
        runtime.futexsleep(0x101f8a08, 0x0, 0xffffffffffffffff)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/os_linux.go:44 +0x3c
        runtime.notesleep(0x101f8a08)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/lock_futex.go:151 +0xb8
        runtime.stopm()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/proc.go:1928 +0xb8
        runtime.startlockedm(0xc0000c1800)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/proc.go:2098 +0x78
        runtime.schedule()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/proc.go:2561 +0x60
        runtime.exitsyscall0(0xc0000c2480)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/proc.go:3135 +0x140
        runtime.mcall(0xc0000c2480)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/asm_ppc64x.s:202 +0x58
        
        goroutine 1 [chan receive]:
        main.CgoCallbackGC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:90 +0x134
        main.main()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/main.go:34 +0x198
        
        goroutine 17 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 19 [syscall]:
        os/signal.signal_recv(0x0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/sigqueue.go:139 +0xf8
        os/signal.loop()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/os/signal/signal_unix.go:23 +0x24
        created by os/signal.init.0
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/os/signal/signal_unix.go:29 +0x3c
        
        goroutine 163 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 162 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 161 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 159 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 158 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 157 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 156 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 155 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 154 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 164 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 185 [runnable]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 153 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 150 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 146 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 145 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 144 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 143 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 142 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 141 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 138 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 136 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 133 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 132 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 130 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 128 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 123 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 184 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 182 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 181 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 180 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 179 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 178 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 177 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 176 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 175 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 174 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 173 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 172 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 170 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 169 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 168 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 167 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 165 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 152 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 186 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 187 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 188 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 189 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 190 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 191 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 192 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 193 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 194 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 195 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 196 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 197 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 198 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 199 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 200 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 201 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 202 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 203 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 204 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 205 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 206 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 207 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 208 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 209 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 210 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 211 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 212 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 213 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 214 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 215 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 216 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 217 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 218 [syscall]:
        main._Cfunc_foo()
        	_cgo_gotypes.go:211 +0x40
        main.CgoCallbackGC.func2(0xc0000700c0)
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:85 +0x24
        created by main.CgoCallbackGC
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:84 +0x104
        
        goroutine 226 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 231 [runnable, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 232 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 241 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 242 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 245 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 246 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 247 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 248 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 249 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 250 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 254 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 257 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 258 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 259 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 260 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 261 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 262 [running, locked to thread]:
        	goroutine running on other thread; stack unavailable
        
        goroutine 263 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 264 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 265 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 266 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 267 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 268 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 269 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 270 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 272 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 273 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 274 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 275 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 276 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 279 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 280 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 281 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 282 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 283 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 284 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 285 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 286 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 287 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 288 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 289 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 291 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 292 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 293 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 295 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:44 +0x58
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 296 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 297 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 298 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 299 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 300 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 301 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 302 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 303 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 304 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 305 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 306 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 307 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 308 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 309 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 310 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 311 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 312 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 313 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 314 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 315 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 316 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 317 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 318 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 319 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 320 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 321 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 322 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 323 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        goroutine 324 [semacquire, locked to thread]:
        runtime.GC()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1064 +0x68
        main.go_callback()
        	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/testdata/testprogcgo/callback.go:42 +0x24
        main._cgoexpwrap_4097c8f6dac3_go_callback()
        	_cgo_gotypes.go:403 +0x24
        
        r0   0xdd	r1   0x7fffcfae7b08
        r2   0x80	r3   0x101f8a08
        r4   0x80	r5   0x0
        r6   0x0	r7   0x0
        r8   0x0	r9   0x0
        r10  0x0	r11  0x0
        r12  0x0	r13  0x7fff7ff8a810
        r14  0x1000c254	r15  0xc0000be678
        r16  0x101112ed	r17  0xc000023340
        r18  0x7fffcfae7950	r19  0x0
        r20  0xc00009a018	r21  0x101f88c0
        r22  0xc00026ef38	r23  0x0
        r24  0x30	r25  0x8
        r26  0x0	r27  0x0
        r28  0x7fff7ff80000	r29  0x7fff7ff7f6f8
        r30  0x101f80e0	r31  0x1000c0e8
        pc   0x1005d49c	ctr  0x0
        link 0x1002b38c	xer  0x0
        ccr  0x54400882	trap 0xc00
FAIL
FAIL	runtime	85.907s

CC @aclements @mknyszek
@bcmills bcmills added Testing NeedsInvestigation labels May 14, 2019
@bcmills bcmills added this to the Go1.13 milestone May 14, 2019
@bcmills bcmills changed the title runtime: TestCgoCallbackGC flake on linux-ppc64le-power9osu runtime: TestCgoCallbackGC flake on linux-ppc64le-power9osu builder May 14, 2019
@andybons andybons removed this from the Go1.13 milestone Jul 8, 2019
@andybons andybons added this to the Go1.14 milestone Jul 8, 2019
@rsc rsc removed this from the Go1.14 milestone Oct 9, 2019
@rsc rsc added this to the Backlog milestone Oct 9, 2019
@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 2, 2020 •
edited

2020-02-25T20:23:54-ba093c4/freebsd-arm64-dmgk
2019-11-23T00:06:41-f29e53d/linux-ppc64le-power9osu
2019-11-12T21:02:24-00e14af/freebsd-arm64-dmgk
2019-11-07T18:06:59-ee2268c/freebsd-arm64-dmgk
2019-09-18T20:35:15-827584e/linux-ppc64le-power9osu

CC @ianlancetaylor

@bcmills bcmills changed the title runtime: TestCgoCallbackGC flake on linux-ppc64le-power9osu builder runtime: occasional hangs in TestCgoCallbackGC Mar 2, 2020
@bcmills bcmills mentioned this issue Mar 2, 2020
Closed
@laboger
Copy link
Contributor

@laboger laboger commented Mar 2, 2020

I noticed that the most recent failure on the power9 builder has a suspicious output file. It is missing the normal header information from the buildlet.

It should be:

linux-ppc64le-power9osu at e674972e8c8ee37dda425e4d294a196395d80483

:: Running /workdir/go/src/make.bash with args ["/workdir/go/src/make.bash"] and env ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOSTNAME=power_02" "GO_BUILDER_ENV=host-linux-ppc64le-power9-osu" "GO_BUILD_KEY_PATH=/buildkey/gobuildkey" "GO_BUILD_KEY_DELETE_AFTER_READ=true" "DEBIAN_FRONTEND=noninteractive" "GOROOT_BOOTSTRAP=/workdir/go1.4" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=300ms" "GO_STAGE0_DL_DELAY=300ms" "WORKDIR=/workdir" "GO_BUILDER_NAME=linux-ppc64le-power9osu" "GO_BUILDER_FLAKY_NET=1" "GOROOT_BOOTSTRAP=/usr/local/go-bootstrap" "GOPPC64=power9" "GOBIN=" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache" "GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"] in dir /workdir/go/src

Building Go cmd/dist using /usr/local/go-bootstrap. (go1.11.1 linux/ppc64le)
Building Go toolchain1 using /usr/local/go-bootstrap.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/ppc64le.
---
Installed Go for linux/ppc64le in /workdir/go
Installed commands in /workdir/go/bin

##### Testing packages.
ok  	archive/tar	0.035s
ok  	archive/zip	0.443s

But instead shows:

linux-ppc64le-power9osu at f29e53d66419713f011cda50a3001cc20950bc7e


##### Testing packages.
ok  	archive/tar	0.074s
ok  	archive/zip	0.374s
...

Almost like it didn't rebuild the toolchain before testing the packages?

@bcmills
Copy link
Member Author

@bcmills bcmills commented May 1, 2020

2020-05-01T05:25:54-e1d1684/freebsd-arm64-dmgk
2020-04-24T23:58:38-396833c/linux-ppc64le-power9osu
2020-04-13T20:38:15-300ed43/netbsd-386-9_0
2020-03-20T00:27:02-a0917eb/freebsd-386-11_2
2020-03-02T23:14:42-4f98932/darwin-amd64-10_14
2020-03-02T16:12:20-ab7ecea/darwin-amd64-race

@bcmills
Copy link
Member Author

@bcmills bcmills commented Jan 27, 2021

2021-01-27T00:15:06-0f797f1/linux-arm64-aws
2020-12-09T03:01:58-6fa06d9/freebsd-arm64-dmgk
2020-10-23T23:01:52-64dc25b/linux-arm64-aws
2020-10-21T09:34:39-9848e93/linux-ppc64le-buildlet
2020-10-14T18:05:16-9c56300/freebsd-arm64-dmgk
2020-09-23T19:51:27-c9551f9/linux-ppc64le-power9osu
2020-09-23T11:49:55-4d7abd7/freebsd-arm64-dmgk
2020-09-22T22:47:43-7f71846/freebsd-arm64-dmgk

@bcmills
Copy link
Member Author

@bcmills bcmills commented Feb 22, 2021

2021-02-19T19:28:53-06b86e9/linux-ppc64le-power9osu
2021-02-09T18:40:13-e9c9683/freebsd-arm64-dmgk

@bcmills bcmills mentioned this issue Feb 25, 2021
Open
@bcmills
Copy link
Member Author

@bcmills bcmills commented May 12, 2021

Looks like sometimes the subprocess does not exit quickly enough when the SIGQUIT is delivered, resulting in a different failure mode (but the same hang):

2021-05-12T15:23:09-0388670/illumos-amd64
2021-05-03T01:35:44-2c9f5a1/illumos-amd64
2021-04-29T19:04:31-87e4dcd/illumos-amd64
2021-04-29T14:06:55-d80da19/illumos-amd64
2021-04-23T21:43:04-79f3513/illumos-amd64
2021-04-23T21:42:57-c0e1301/illumos-amd64
2021-04-23T21:42:56-073f913/illumos-amd64
2021-04-20T22:47:50-48e3d92/solaris-amd64-oraclerel
2021-04-14T04:02:01-6d8ba77/illumos-amd64
2021-04-06T22:55:58-b084073/illumos-amd64
2021-04-02T19:45:52-41cf18e/illumos-amd64
2021-03-16T13:06:17-120b9eb/illumos-amd64
2021-03-14T21:10:45-3cdd5c3/illumos-amd64
2021-03-11T05:38:13-3a3b816/linux-ppc64le-buildlet
2021-03-07T23:58:09-597b5d1/illumos-amd64
2021-03-05T17:46:57-67b9ecb/illumos-amd64
2021-02-19T00:40:16-985d087/linux-ppc64le-buildlet
2021-02-19T00:04:49-a1222b7/linux-ppc64le-buildlet
2021-02-19T00:01:59-229695a/linux-ppc64le-buildlet
2020-12-29T21:49:26-780b4de/linux-ppc64le-buildlet
2020-10-30T18:05:46-34665c6/solaris-amd64-oraclerel
2019-07-27T23:29:18-a25c287/solaris-amd64-smartosbuildlet
2019-05-24T09:38:37-c468ad0/solaris-amd64-smartosbuildlet
2019-05-23T01:38:26-7a567a6/solaris-amd64-smartosbuildlet
2019-05-22T20:41:27-983986f/solaris-amd64-smartosbuildlet
2019-05-14T18:48:16-6081a9f/solaris-amd64-smartosbuildlet
2019-05-13T13:54:54-c583104/solaris-amd64-smartosbuildlet
2019-05-09T21:13:25-b4a472b/solaris-amd64-smartosbuildlet
2019-05-09T17:11:16-a44c3ed/solaris-amd64-smartosbuildlet
2019-05-07T12:48:04-a88cb1d/solaris-amd64-smartosbuildlet
2019-05-05T08:09:30-12e6322/solaris-amd64-smartosbuildlet

@bcmills
Copy link
Member Author

@bcmills bcmills commented Oct 11, 2021

greplogs --dashboard -md -l -e '(?ms)FAIL: TestCgoCallbackGC.*SIGQUIT' --since=2021-02-20

2021-10-11T16:48:22-aa1c6f5/linux-ppc64le-power9osu
2021-09-27T20:56:37-f961d8e/linux-ppc64le-power9osu
2021-09-17T22:19:21-3fa35b5/linux-ppc64le-power9osu
2021-09-15T03:29:41-b26d325/linux-ppc64le-power9osu
2021-08-28T01:49:50-f371b30/linux-ppc64le-power9osu
2021-08-26T19:07:27-1f8d456/linux-ppc64le-power9osu
2021-08-16T17:24:37-850768b/linux-ppc64le-power9osu
2021-08-16T13:39:01-02f932e/linux-ppc64le-power9osu
2021-08-16T07:44:47-160d797/linux-ppc64le-power9osu
2021-07-27T20:52:37-c8cf0f7/linux-ppc64le-power9osu
2021-07-26T21:34:18-9c81fd5/freebsd-arm64-dmgk
2021-04-29T13:23:02-47cb0c4/linux-ppc64le-power9osu
2021-04-27T21:16:55-30002e6/linux-ppc64le-power9osu
2021-04-14T16:18:34-ad44dfb/linux-arm64-aws
2021-02-25T01:30:37-ff614b1/netbsd-386-9_0
2021-02-23T21:18:30-fa40c02/linux-ppc64le-power9osu
2021-02-23T19:55:03-a671e33/linux-ppc64le-power9osu

@laboger
Copy link
Contributor

@laboger laboger commented Oct 27, 2021

I have a hunch that this change https://go-review.googlesource.com/c/go/+/339989 might fix the problem.
If I look at the traceback dump in the ppc64lle log, it appears that it trying to treat a C stack as a Go stack.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Dec 1, 2021

I have a hunch that this change https://go-review.googlesource.com/c/go/+/339989 might fix the problem.

Unfortunately it does not seem to have. (That change was merged Oct. 26.)

greplogs --dashboard -md -l -e '(?ms)FAIL: TestCgoCallbackGC.*SIGQUIT' --since=2021-10-28

2021-11-30T20:04:58-7ccbcc9/linux-ppc64le-buildlet
2021-11-16T07:47:15-6b3f4d3/freebsd-arm64-dmgk
2021-11-05T00:52:04-1c4cfd8/linux-ppc64le-power9osu
2021-10-29T19:30:03-4592933/linux-ppc64le-power9osu

@bcmills
Copy link
Member Author

@bcmills bcmills commented Dec 15, 2021

greplogs --dashboard -md -l -e '(?ms)FAIL: TestCgoCallbackGC.*SIGQUIT' --since=2021-12-02

2021-12-13T16:42:31-4b3d8d1/linux-ppc64le-power9osu
2021-12-13T15:48:26-f4ca598/linux-arm64-aws

@bcmills
Copy link
Member Author

@bcmills bcmills commented Dec 15, 2021

This is still ongoing in the builders — and note that linux-arm64 is a first-class port — so marking as release-blocker via #11811.

@bcmills bcmills added okay-after-beta1 release-blocker labels Dec 15, 2021
@bcmills bcmills removed this from the Backlog milestone Dec 15, 2021
@bcmills bcmills added this to the Go1.18 milestone Dec 15, 2021
@bcmills bcmills removed the okay-after-beta1 label Dec 15, 2021
@cherrymui
Copy link
Member

@cherrymui cherrymui commented Dec 21, 2021

Could it be that it is simply slow? It runs 200 GCs, while the system may be under load running other tests (it has t.Parallel()) and other processes in all.bash. Note that the test checks RUNTIME_TESTING_SHORT whereas the test runner actually sets RUNTIME_TEST_SHORT=1 https://cs.opensource.google/go/go/+/master:src/runtime/crash_test.go;l=70 (note the different spelling, TESTING vs. TEST.)

@gopherbot
Copy link

@gopherbot gopherbot commented Dec 21, 2021

Change https://golang.org/cl/373614 mentions this issue: runtime: check the correct environment variable in TestCgoCallbackGC

gopherbot pushed a commit that referenced this issue Dec 21, 2021
@cherrymui
runtime: check the correct environment variable in TestCgoCallbackGC
e087949 
The test checks RUNTIME_TESTING_SHORT, whereas the test runner
actually set RUNTIME_TEST_SHORT. Check the correct one.

Updates #32023.

Change-Id: Ie8ab00e1f5b8c02112a9aa1ee0e56028185c8a44
Reviewed-on: https://go-review.googlesource.com/c/go/+/373614
Trust: Cherry Mui <cherryyz@google.com>
Run-TryBot: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@aclements
Copy link
Member

@aclements aclements commented Jan 12, 2022

This hasn't happened again since 2021-12-13, but unfortunately it's rare enough that that doesn't give us much signal on whether Cherry's change fixed it.

@aclements
Copy link
Member

@aclements aclements commented Jan 12, 2022

Given how rare this is and the likelihood that this test was just slow and we accidentally weren't filtering it out, I'm going to drop release-blocker.

@aclements aclements removed this from the Go1.18 milestone Jan 12, 2022
@aclements aclements added this to the Unplanned milestone Jan 12, 2022
@bcmills bcmills added the WaitingForInfo label Jan 12, 2022
@bcmills
Copy link
Member Author

@bcmills bcmills commented Jan 12, 2022

Given the plausible fix, should we go ahead and close the issue until/unless it recurs?

@aclements
Copy link
Member

@aclements aclements commented Jan 12, 2022

Sounds good to me.

@bcmills
Copy link
Member Author

@bcmills bcmills commented May 13, 2022 •
edited

greplogs -l -e 'FAIL: TestCgoCallbackGC .*(?:\n[ ]+.*)*SIGQUIT' --since=2021-12-14
2022-05-12T19:04:52-f269d90/netbsd-386-9_0

--- FAIL: TestCgoCallbackGC (60.08s)
    testenv.go:460: [/tmp/go-build4271962090/testprogcgo.exe CgoCallbackGC] exit status: exit status 2
    crash_cgo_test.go:73: expected "OK\n", but got:
        SIGQUIT: quit
        PC=0x80ad4c7 m=22 sigcode=0

(CC @golang/runtime)

@bcmills bcmills reopened this May 13, 2022
@bcmills bcmills removed the WaitingForInfo label May 13, 2022
@bcmills bcmills removed this from the Unplanned milestone May 13, 2022
@bcmills bcmills added this to the Backlog milestone May 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Testing
Projects
Go Runtime
Status: Todo
Milestone
Backlog
Development

No branches or pull requests
7 participants
@rsc @andybons @aclements @bcmills @gopherbot @laboger @cherrymui