cmd/cgo/internal/test: TestCallbackCallersSEH failures

[qmuntal]

Go version

go version devel go1.22-e9b3ff15f4 Wed Jan 10 17:35:49 2024 +0000 windows/amd64

Output of go env in your module/workspace:

set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\aaa\AppData\Local\go-build
set GOENV=C:\Users\aaa\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\aaa\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\aaa\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Users\aaa\code\golang-go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLCHAIN=auto
set GOTOOLDIR=C:\Users\aaa\code\golang-go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=devel go1.22-e9b3ff15f4 Wed Jan 10 17:35:49 2024 +0000
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\aaa\code\golang-go\src\cmd\go.mod
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=C:\Users\aaa\AppData\Local\Temp\go-build998051253=/tmp/go-build -gno-record-gcc-switches

What did you do?

go test --ldflags=-linkmode=internal -run=TestCallbackCallersSEH ./cmd/cgo/internal/test

What did you see happen?

--- FAIL: TestCallbackCallersSEH (0.00s)
    callback_windows.go:107: incorrect backtrace:
        want:   [test._Cfunc_backtrace test.testCallbackCallersSEH.func1.1 test.testCallbackCallersSEH.func1 test.goCallback test._Cfunc_callback test.nestedCall.func1 test.nestedCall test.testCallbackCallersSEH test.TestCallbackCallersSEH]
        got:    []
FAIL
FAIL    cmd/cgo/internal/test   2.527s
FAIL

What did you expect to see?

=== RUN   TestCallbackCallersSEH
--- PASS: TestCallbackCallersSEH (0.00s)
PASS
ok      cmd/cgo/internal/test   2.286s

Some more context:

• Only reproduces when using the Go internal linker.
• The test fails only with certain gcc versions. I've reproduced the failure with:

>gcc --version
gcc (x86_64-posix-seh-rev2, Built by MinGW-W64 project) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[qmuntal]

I've tracked down the failure to an issue with how the .pdata entries are laid on disk. The docs says that these entries should be sorted using the function start address, but the Go linker not always honor this when copying the .pdata entries from the host objects, which makes the Windows runtime to find the wrong entry, therefore producing an incorrect unwinding.

Details of how the unsorted case happens: the Go linker concatenates .pdata sections from host objects without modifying them, relying on the host compiler to produce sorted entries (which they do). The host objects can contain unreachable function with their corresponding .pdata entries, which will eventually be replaced by the Go linker for runtime.unreachableMethod. As the unreachable function no longer exists, the Go linker also naively updates start address of the corresponding .pdata entry to runtime.unreachableMethod, when what it should do to keep the .pdata entries sorted is remove that entry. Unfortunately, this is not straightforward because .pdata sections from host objects are copied as a whole, not as individual entries.

It is already too late to fix this issue for go1.22, considering that the release is in ~3 weeks. We could disable cgo stack unwinding when using internal linking and reenable it for go1.23, but I would suggest to just skip TestCallbackCallersSEH as flaky until the issue is fixed. This feature has been implemented in go1.22, before it was not working at all, so having something that mostly works is better than nothing. Also, some unwinders, like WinDbg, can handle unsorted .pdata entries and show the correct call stack.

@golang/release @golang/windows

[cherrymui]

Thanks for the analysis.

The host objects can contain unreachable function with their corresponding .pdata entries, which will eventually be replaced by the Go linker for runtime.unreachableMethod.

Interesting. The intention of runtime.unreachableMethod was for Go functions. I didn't realize this could happen. One possibility is not rewriting the relocations in host objects, leaving it to point to, say, 0.

I agree that for the moment skipping the test is good. We could skip just in internal linking mode. I think cmd/dist (all.bat) passes internal build tag when running them in internal linking mode. Users generally don't explicitly use internal linking mode, so unlikely to be affected.

[gopherbot]

Change https://go.dev/cl/556635 mentions this issue: cmd/cgo/internal/test: skip TestCallbackCallersSEH when internal linking

[gopherbot]

Change https://go.dev/cl/557815 mentions this issue: [release-branch.go1.22] cmd/cgo/internal/test: skip TestCallbackCallersSEH when internal linking