Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: Race detector causes exit status 0xc0000139 on Windows 11 with 1.21rc2 running gcc 9.2.0 #61058

Closed
jameshartig opened this issue Jun 29, 2023 · 8 comments
Assignees
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. Documentation NeedsFix The path to resolution is known, but the work has not been done. OS-Windows release-blocker
Milestone

Comments

@jameshartig
Copy link
Contributor

What version of Go are you using (go version)?

$ go version
go version go1.21rc2 windows/amd64

Does this issue reproduce with the latest release?

Reproducible with 1.21rc2 but not reproducible with 1.20.5. I'm not sure how to download 1.21rc1 to test that.

What operating system and processor architecture are you using (go env)?

Windows 11 Pro Version 10.0.22621 Build 22621

go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\james\AppData\Local\go-build
set GOENV=C:\Users\james\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\james\go\pkg\mod
set GONOPROXY=gerrit.levenlabs.com,gitlab.com/levenlabs
set GONOSUMDB=gerrit.levenlabs.com,gitlab.com/levenlabs
set GOOS=windows
set GOPATH=C:\Users\james\go;
set GOPRIVATE=gerrit.levenlabs.com,gitlab.com/levenlabs
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLCHAIN=auto
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.21rc2
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\james\Dropbox\aftermath\backend\go-llib\go.mod
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=C:\Users\james\AppData\Local\Temp\go-build2611928154=/tmp/go-build -gno-record-gcc-switches

I had previously installed tdm-gcc 9.2.0 which causes the error:

gcc -v Output
> gcc -v
Using built-in specs.
COLLECT_GCC=C:\TDM-GCC-64\bin\gcc.exe
COLLECT_LTO_WRAPPER=C:/TDM-GCC-64/bin/../libexec/gcc/x86_64-w64-mingw32/9.2.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../../../src/gcc-git-9.2.0/configure --build=x86_64-w64-mingw32 --enable-targets=all --enable-languages=ada,c,c++,fortran,lto,objc,obj-c++ --enable-libgomp --enable-lto --enable-graphite --enable-cxx-flags=-DWINPTHREAD_STATIC --disable-build-with-cxx --disable-build-poststage1-with-cxx --enable-libstdcxx-debug --enable-threads=posix --enable-version-specific-runtime-libs --enable-fully-dynamic-string --enable-libstdcxx-threads --enable-libstdcxx-time --with-gnu-ld --disable-werror --disable-nls --disable-win32-registry --enable-large-address-aware --disable-rpath --disable-symvers --prefix=/mingw64tdm --with-local-prefix=/mingw64tdm --with-pkgversion=tdm64-1 --with-bugurl=http://tdm-gcc.tdragon.net/bugs
Thread model: posix
gcc version 9.2.0 (tdm64-1) 

But upon upgrading to tdm-gcc 10.3.0 the error no longer occurs:

gcc -v Output
> gcc -v
Using built-in specs.
COLLECT_GCC=C:\TDM-GCC-64\bin\gcc.exe
COLLECT_LTO_WRAPPER=C:/TDM-GCC-64/bin/../libexec/gcc/x86_64-w64-mingw32/10.3.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../../../src/gcc-git-10.3.0/configure --build=x86_64-w64-mingw32 --enable-targets=all --enable-languages=ada,c,c++,fortran,jit,lto,objc,obj-c++ --enable-libgomp --enable-lto --enable-graphite --enable-cxx-flags=-DWINPTHREAD_STATIC --disable-build-with-cxx --disable-build-poststage1-with-cxx --enable-libstdcxx-debug --enable-threads=posix --enable-version-specific-runtime-libs --enable-fully-dynamic-string --enable-libstdcxx-filesystem-ts=yes --disable-libstdcxx-pch --enable-libstdcxx-threads --enable-libstdcxx-time=yes --enable-mingw-wildcard --with-gnu-ld --disable-werror --enable-nls --disable-win32-registry --enable-large-address-aware --disable-rpath --disable-symvers --prefix=/mingw64tdm --with-local-prefix=/mingw64tdm --with-pkgversion=tdm64-1 --with-bugurl=https://github.com/jmeubank/tdm-gcc/issues
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.3.0 (tdm64-1)

What did you do?

Trying to run any tests with -race and gcc 9.2.0 ends up failing with exit status 0xc0000139.
https://go.dev/play/p/RFjBrSPywGg

> go.exe test ./... -v -race
exit status 0xc0000139
> go.exe test ./... -v
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
PASS

But upon upgrading to gcc 10.3.0:

> go.exe test ./... -v -race
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
PASS

What did you expect to see?

I expected for the race detector not to break between versions without something in the release notes pointing out an expectation of breakage. All I noticed in the release notes for 1.21 that might be related is:

On Windows AMD64, the linker (with help from the compiler) now emits SEH unwinding data by default, which improves the integration of Go applications with Windows debuggers and other tools.

But it's not obvious that this would require an upgraded gcc. It wasn't until after several hours of Googling and debugging that I ended up attempting to update gcc.

What did you see instead?

Instead I saw a bizarre exit status that led me down a deep rabbit hole on Google to try and figure out why my go tests suddenly don't work.

@jameshartig
Copy link
Contributor Author

After reading through #35006 it seems like the recommendation is to use llvm. Downloading the latest ucrt from https://github.com/mstorsjo/llvm-mingw/releases and putting that in my path seems to also be working.

@bcmills bcmills changed the title testing: Race detector causes exit status 0xc0000139 on Windows 11 with 1.21rc2 running gcc 9.2.0 runtime: Race detector causes exit status 0xc0000139 on Windows 11 with 1.21rc2 running gcc 9.2.0 Jun 29, 2023
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jun 29, 2023
@bcmills
Copy link
Contributor

bcmills commented Jun 29, 2023

Given that the binary works with a newer GCC, this sounds like a GCC bug rather than a Go bug.

That said, if we know of specific compilers that do or do not work with the race detector, or with Go in general, then probably we should add that detail to https://go.dev/wiki/MinimumRequirements and/or https://go.dev/doc/articles/race_detector#Requirements.

(CC @golang/windows, @golang/runtime)

@cagedmantis cagedmantis added this to the Backlog milestone Jun 29, 2023
@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 29, 2023
@qmuntal
Copy link
Contributor

qmuntal commented Jun 29, 2023

GCC 9.2 is still widely used, we should understand what happened before the final go1.22 release, I'll investigate if SEH or the frame pointer changes have anything to do with this issue.

@qmuntal
Copy link
Contributor

qmuntal commented Jun 30, 2023

Did some archeology and found the culprit: the program fails to load the WakeByAddressSingle, WakeByAddressAll and WaitOnAddress symbols from kernel32.dll. These symbols are required by the new race_windows_amd64.syso library, updated in go1.21 as part of #49761, specifically in CL 420197.

The issue is that the missing symbols are not part of kernel32.dll since Windows 10, they are exported in libsynchronization.a instead. Mingw-w64 fixed this loading issue in v8.0.0 (mingw-w64/mingw-w64@68d0e7d), but tdm-gcc (which repackage mingw-w64) didn't port that change till v10 (jmeubank/mingw-w64@68d0e7d). Therefore, tdm-gcc v9 can't be used with go1.21.

That said, if we know of specific compilers that do or do not work with the race detector, or with Go in general, then probably we should add that detail to https://go.dev/wiki/MinimumRequirements and/or https://go.dev/doc/articles/race_detector#Requirements.

This was discussed and rejected in #35006. This comment made by @thanm gives a good reasoning: #35006 (comment).

@jameshartig
Copy link
Contributor Author

@qmuntal thank you for the in-depth analysis and explanation! Does it make sense to mention this in the release notes at all if GCC 9 is still fairly popular?

@qmuntal
Copy link
Contributor

qmuntal commented Jul 1, 2023

Does it make sense to mention this in the release notes at all if GCC 9 is still fairly popular?

This is not a problem with GCC or Mingw-w64 9, they are both fine, but with tdm-gcc 9. I don't know how popular is tdm-gcc and if it worth adding this issue to the release notes, as add this kind of things in there.

@mknyszek mknyszek modified the milestones: Backlog, Go1.21 Jul 5, 2023
@mknyszek mknyszek added NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 5, 2023
@bcmills
Copy link
Contributor

bcmills commented Jul 11, 2023

This was discussed and rejected in #35006. This comment made by @thanm gives a good reasoning: #35006 (comment).

I agree with that reasoning, but https://go.dev/wiki/MinimumRequirements#cgo is currently pretty unambiguous that “gcc 4.6 or newer is required”, implying that every version higher than that one is supported. I don't think it's accurate at this point.

I think what we really mean is “cmd/go will pass C compiler flags compatible with gcc 4.6 or newer”, which is a much weaker statement. I'm not sure exactly how to update the wiki page to reflect that.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/508461 mentions this issue: _content/doc/articles/race_detector: document windows C compiler version requirements

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. labels Jul 17, 2023
antoninbas added a commit to antoninbas/antrea that referenced this issue Aug 10, 2023
After upgrading to Go v1.21, Go Windows unit tests (which we run with
the race detector enabled) cannot be run: go exists immediately with
`exit status 0xc0000139`.

That may be caused by older mingw-w64 runtime libraries. See
golang/go#61058.

Signed-off-by: Antonin Bas <abas@vmware.com>
antoninbas added a commit to antoninbas/antrea that referenced this issue Aug 21, 2023
After upgrading to Go v1.21, Go Windows unit tests (which we run with
the race detector enabled) cannot be run: go exists immediately with
`exit status 0xc0000139`.

That may be caused by older mingw-w64 runtime libraries. See
golang/go#61058.

Signed-off-by: Antonin Bas <abas@vmware.com>
antoninbas added a commit to antoninbas/antrea that referenced this issue Aug 22, 2023
After upgrading to Go v1.21, Go Windows unit tests (which we run with
the race detector enabled) cannot be run: go exists immediately with
`exit status 0xc0000139`.

That may be caused by older mingw-w64 runtime libraries. See
golang/go#61058.

Signed-off-by: Antonin Bas <abas@vmware.com>
antoninbas added a commit to antrea-io/antrea that referenced this issue Aug 22, 2023
* Replace usage of "golang.org/x/exp/slices" with new "slices" package.
* Remove calls to "rand.Seed": starting with Go v1.20, the generator is seeded
  randomly at program startup, and we no longer need to do it manually.
* The codegen image has been updated to kubernetes-1.26.4-build.0 to
  include the updated version of Go. No changes im generated files.
* Stop using "golang.org/x/exp/rand" in test/e2e/performance_test.go: this is
  unrelated to the Go update.

* Use windows-2022 runners for Go Windows unit tests

After upgrading to Go v1.21, Go Windows unit tests (which we run with
the race detector enabled) cannot be run: go exists immediately with
`exit status 0xc0000139`.

That may be caused by older mingw-w64 runtime libraries. See
golang/go#61058.

* Fix Dockerfile for integrations tests image

The Dockerfile should install the latest patch release for the minor Go
version currently supported by Antrea.

Fixes #5371

Signed-off-by: Antonin Bas <abas@vmware.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. Documentation NeedsFix The path to resolution is known, but the work has not been done. OS-Windows release-blocker
Projects
None yet
Development

No branches or pull requests

8 participants