Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime/race: ThreadSanitizer failed to allocate 0x0000005c9000 (6066176) bytes at 0x200dc940a0000 (error code: 87) #46099

Closed
neclepsio opened this issue May 11, 2021 · 51 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows
Milestone

Comments

@neclepsio
Copy link

neclepsio commented May 11, 2021

What version of Go are you using (go version)?

$ go version
go version go1.16.4 windows/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\---\AppData\Local\go-build
set GOENV=C:\Users\---\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=c:\---\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=c:\---
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\---\Go\go1.16.4
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\---\Go\go1.16.4\pkg\tool\windows_amd64
set GOVERSION=go1.16.4
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\---\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\---\AppData\Local\Temp\go-build71564379=/tmp/go-build -gno-record-gcc-switches

What did you do?

Use the source in "Usage" section of https://github.com/go-gl/glfw, then run:

$ go build -race; ./test

What did you expect to see?

The program correctly running, as if -race was not provided.

What did you see instead?

ERROR: ThreadSanitizer failed to allocate 0x0000005c9000 (6066176) bytes at 0x200dc940a0000 (error code: 87)

The error appears on each run. The count, address and error code are always the same. Using different program, the count and address are different.

@neclepsio neclepsio changed the title race detector: ThreadSanitizer failed to allocate 0x000000c01000 (12587008) bytes at 0x200de726e0000 (error code: 87) race detector: ThreadSanitizer failed to allocate 0x0000005c9000 (6066176) bytes at 0x200dc940a0000 (error code: 87) May 11, 2021
@randall77
Copy link
Contributor

The race detector uses ~10x the memory that a run without -race uses. Are you actually running out of memory? How much memory does a regular run use?

It is possible that the race detector can't allocate memory because the address space it wants is being used by something else.
I'm not sure exactly what would cause that, but if your code mmaps or dlopens lots of stuff, that might be a contributing factor.

@neclepsio
Copy link
Author

The task manager show less then 10Mb of use for the simple example. Looking to GLFW source code, mmap is used for some small bitmaps and there are some dlopen, just to OpenGL and related libraries, but it seems to me that they are not used in Windows (LoadLibraryA is used instead, I don't know if it is the same). Moreover, error code 87 is "the parameter is incorrect" under Windows.

@randall77
Copy link
Contributor

Hm, I don't know then. You have officially overreached my Windows knowledge :(

@heschi heschi added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 11, 2021
@heschi heschi changed the title race detector: ThreadSanitizer failed to allocate 0x0000005c9000 (6066176) bytes at 0x200dc940a0000 (error code: 87) runtime/race: ThreadSanitizer failed to allocate 0x0000005c9000 (6066176) bytes at 0x200dc940a0000 (error code: 87) May 11, 2021
@heschi heschi added this to the Backlog milestone May 11, 2021
@heschi
Copy link
Contributor

heschi commented May 11, 2021

cc @dvyukov @bufflig

@jazzy-crane
Copy link

I too am seeing this on my project. I need to confirm, but I think I wasn't seeing the issue with 1.16.3, but then hit it with 1.16.4

@targodan
Copy link

targodan commented May 14, 2021

Also seeing this with fkie-cad/yapscan on windows only.
That being said, I do do a bunch of c-stuff and reading remote process memory.

go version go1.16.4 windows/amd64

@jazzy-crane
Copy link

I too am seeing this on my project. I need to confirm, but I think I wasn't seeing the issue with 1.16.3, but then hit it with 1.16.4

Sorry, scratch that. I do see with 1.16.3 too. Go version bump coincided with other toolchain upgrades

@networkimprov
Copy link

cc @zx2c4

@neclepsio
Copy link
Author

The problem is still present in 1.16.5.

@neclepsio
Copy link
Author

Also affects 1.17beta1.

@neclepsio
Copy link
Author

Still present in 1.17rc1

@networkimprov
Copy link

cc @dvyukov

@dvyukov
Copy link
Member

dvyukov commented Jul 22, 2021

I don't have a windows machine, so I can't debug it.
Why can VirtualAlloc return 87? I thought maybe it's because size 0x5c9000 is not a multiple of allocation granularity (64k), but the msdn page says that in such cases the size is simply rounded up.

@targodan
Copy link

Is the MEM_LARGE_PAGES flag used? In that case the size needs to be a multiple of GetLargePageMinimum.

  1. Include the MEM_LARGE_PAGES value when calling the VirtualAlloc function. The size and alignment must be a multiple of the large-page minimum.

Source: https://docs.microsoft.com/en-us/windows/win32/memory/large-page-support

@neclepsio
Copy link
Author

I don't know if it's relevant, but according to this message in haskell forum: "this means that address given to VirtualAlloc is either not reserved yet, or that size is too big (ie the block-to-be-committed isn't inside one VirtualAlloc MEM_RESERVE block)". Another source point to ASLR, but this should not be the case since the problem is always with the same address. This IBM bug report states: Windows API VirtualAlloc is requesed to allocate memory ofsize 64KB with flag MEM_LARGE_PAGES. This is a non-standard allocation and the API fails with an error "The parameter is incorrect".

@dvyukov
Copy link
Member

dvyukov commented Jul 22, 2021

MEM_LARGE_PAGES does not seem to be used in compiler-rt/* dir:
https://github.com/llvm/llvm-project/search?q=MEM_LARGE_PAGES

@neclepsio
Copy link
Author

neclepsio commented Jul 22, 2021

This function matches the error message. There is an exception for Windows AddressSanitizer not applied to Go but it seems not related.

@targodan
Copy link

It also seems like the allocation is done with a fixed address?

Even if that is not the core of the issue in this case (since it seems to always happen), wouldn't that potentially lead to pseudorandom failures if ASLR decides to load a DLL there?

@dvyukov
Copy link
Member

dvyukov commented Jul 22, 2021

It also seems like the allocation is done with a fixed address?

Yes, with a fixed address.
Well, first, tsan needs to allocate memory at a fixed address. So it's not that it's done for no reason, nor that it's possible to just remove the address.
Second, on linux tsan will avoid regions where kernel can load anything and it reserves the remaining regions, so that even user mmap's won't happen at these addresses. I don't remember what happen on windows.

@targodan
Copy link

I see. Thanks for clarifying. :)

@AlexRouSg
Copy link
Contributor

AlexRouSg commented Jul 22, 2021

Thought I'd give some data points:

It runs fine if I cross compiled from linux with (GCC) 9.3-posix 20200320
It does not run if I used TDM GCC 9.2 or 10.3
It also runs fine with (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 8.1.0

Looks like it might be specific to TDM GCC

EDIT:

Running with gdb, it crashes at:
racecall(&__tsan_map_shadow, start, size, 0, 0)

https://github.com/golang/go/blob/go1.16.6/src/runtime/race.go#L393
https://github.com/golang/go/blob/go1.16.6/src/runtime/race_amd64.s#L413

@neclepsio
Copy link
Author

This also happens with MSYS2 GCC.

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=c:/Ignazio/Lavoro/IgnPack/Go/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../gcc-10.3.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 
--host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 
--with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/include --libexecdir=/mingw64/lib 
--enable-bootstrap --enable-checking=release --with-arch=x86-64 --with-tune=generic 
--enable-languages=c,lto,c++,fortran,ada,objc,obj-c++,jit --enable-shared --enable-static --enable-libatomic 
--enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-filesystem-ts=yes 
--enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --enable-lto --enable-libgomp 
--disable-multilib --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers 
--with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 
--with-isl=/mingw64 --with-pkgversion='Rev5, Built by MSYS2 project'
 --with-bugurl=https://github.com/msys2/MINGW-packages/issues --with-gnu-as --with-gnu-ld 
--with-boot-ldflags='-pipe -Wl,--dynamicbase,--high-entropy-va,--nxcompat,--default-image-base-high 
-Wl,--disable-dynamicbase -static-libstdc++ -static-libgcc' 'LDFLAGS_FOR_TARGET=-pipe 
-Wl,--dynamicbase,--high-entropy-va,--nxcompat,--default-image-base-high' 
--enable-linker-plugin-flags='LDFLAGS=-static-libstdc++\ -static-libgcc\ -pipe\ 
-Wl,--dynamicbase,--high-entropy-va,--nxcompat,--default-image-base-high\ -Wl,--stack,12582912'
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.3.0 (Rev5, Built by MSYS2 project)

@MuweiHe
Copy link

MuweiHe commented Aug 26, 2021

Having the same issue when trying running make test on azure pipelines. Win 2019 vm, GCC 11.2, Go 1.16.3. Has anyone tried with earlier/newer versions of Go and does that work? And the address each run is different in my case..

@jazzy-crane
Copy link

Having the same issue when trying running make test on azure pipelines. Win 2019 vm, GCC 11.2, Go 1.16.3. Has anyone tried with earlier/newer versions of Go and does that work? And the address each run is different in my case..

I didn't have this problem with 1.16.3 , but was running musl GCC 9.2.1 mingw64 toolchain . When I upgraded Go to 1.16.4 I upgraded the toolchain to GCC 10.X at the same time. That's when I started having problems. So I think the GCC version is the significant thing, not the Go version. Unfortunately I haven't been able to try with >1.16.3 and a 9.2.1 toolchain

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 7, 2022
bep added a commit to bep/hugo that referenced this issue Jul 8, 2022
cfergeau added a commit to cfergeau/crc that referenced this issue Jul 25, 2022
Because of golang/go#46099 'make check' on
Windows in GitHub Actions is failing with '==34372==ERROR:
ThreadSanitizer failed to allocate 0x000003061000 (50728960) bytes at
0x200dabe7d0000 (error code: 87)'

This commit downgrades mingw to 10.0.20 in order to avoid this problem.
This will need to be reverted once this is fixed in mingw or golang.
cfergeau added a commit to cfergeau/crc that referenced this issue Jul 25, 2022
Because of golang/go#46099 'make check' on
Windows in GitHub Actions is failing with '==34372==ERROR:
ThreadSanitizer failed to allocate 0x000003061000 (50728960) bytes at
0x200dabe7d0000 (error code: 87)'

This commit downgrades mingw to 10.0.20 in order to avoid this problem.
This will need to be reverted once this is fixed in mingw or golang.
praveenkumar pushed a commit to crc-org/crc that referenced this issue Jul 26, 2022
Because of golang/go#46099 'make check' on
Windows in GitHub Actions is failing with '==34372==ERROR:
ThreadSanitizer failed to allocate 0x000003061000 (50728960) bytes at
0x200dabe7d0000 (error code: 87)'

This commit downgrades mingw to 10.0.20 in order to avoid this problem.
This will need to be reverted once this is fixed in mingw or golang.
@thanm
Copy link
Contributor

thanm commented Jul 31, 2022

If you are building Go from source, please try the new race detector runtime (pending submit) to see if it resolves this issue. From your Go repo on windows:

 git fetch https://go.googlesource.com/go refs/changes/97/420197/2 && git checkout FETCH_HEAD

This version of the runtime requires a more up-to-date C compiler version (in particular it requires libsynchronization.a).

yashsingh74 pushed a commit to yashsingh74/containerd-main that referenced this issue Jul 31, 2022
There is currently an issue in the race detector in Go on Windows when
used with a newer version of GCC. The issue was first reported here:

golang/go#46099

Fixes containerd#7104

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
(cherry picked from commit 1ef4bda)

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
@neclepsio
Copy link
Author

With no other apparent change in the toolchain, Go 1.19 seems to solve for me.

@Southclaws
Copy link

Southclaws commented Aug 8, 2022

With no other apparent change in the toolchain, Go 1.19 seems to solve for me.

maybe it was this?

image

even though the changes don't affect windows, perhaps some minor dependent change inadvertently fixed the bug...?

dcantah added a commit to dcantah/containerd that referenced this issue Aug 18, 2022
Previously we were downgrading mingw to work around an issue in the race
detector in Go on Windows when used with a newer version of GCC. The
issue was first reported here:

golang/go#46099

Shortly after the release of 1.19 someone had commented this issue was
solved for them, and after trying it out in some test runs on actions
machines, it seems to be the case. The downgrade was mostly harmless
except for two shortcomings:

1. It took quite a while for the package to get downloaded+installed.

2. Chocolatey would frequently fail to download with `The remote file
either doesn't exist, is unauthorized, or is forbidden for url ...
Exception calling "GetResponse" with "0" argument(s): "The request
was aborted: Could not create SSL/TLS secure channel."` Restarting the
failed run would often resolve this, but a 50-50 shot of things working
is not a great situation.

Be nice to find what seemingly fixed the issue, but getting the CI in
working order is a good idea.

Signed-off-by: Daniel Canter <dcanter@microsoft.com>
dcantah added a commit to dcantah/containerd that referenced this issue Aug 18, 2022
This reverts commit 1ef4bda.

Previously we were downgrading mingw to work around an issue in the race
detector in Go on Windows when used with a newer version of GCC. The
issue was first reported here:

golang/go#46099

Shortly after the release of 1.19 someone had commented this issue was
solved for them, and after trying it out in some test runs on actions
machines, it seems to be the case. Disabling ASLR got things in order, and
PIE was disabled for -race builds in 1.19, so this is likely the reason
things work now:
golang/go@0c7fcf6.

The downgrade was mostly harmless except for two shortcomings:

1. It took quite a while for the package to get downloaded+installed.

2. Chocolatey would frequently fail to download with `The remote file
either doesn't exist, is unauthorized, or is forbidden for url ...
Exception calling "GetResponse" with "0" argument(s): "The request
was aborted: Could not create SSL/TLS secure channel."` Restarting the
failed run would often resolve this, but a 50-50 shot of things working
is not a great situation.

Signed-off-by: Daniel Canter <dcanter@microsoft.com>
@dcantah
Copy link

dcantah commented Aug 18, 2022

Also can confirm 1.19 gets things in order again.. Believe it was this set of patches from @thanm 0c7fcf6 eaf2125

@Southclaws
Copy link

Southclaws commented Aug 26, 2022

Updated to 1.19 and new error...

Build Error: go test -c -o d:\Work\odin\odin\api\src\services\deal\__debug_bin.exe -gcflags all=-N -l -v -race .
# runtime/cgo
In file included from c:\program files\x86_64-w64-mingw32-native\lib\gcc\x86_64-w64-mingw32\11.2.1\include-fixed\limits.h:34,
                 from c:\program files\x86_64-w64-mingw32-native\include\stdlib.h:11,
                 from _cgo_export.c:3:
c:\program files\x86_64-w64-mingw32-native\include\syslimits.h:12:25: error: no include path in which to search for limits.h
   12 | #include_next <limits.h>
      |                         ^ (exit status 2)

I just want to debug 1 test 😭

Edit: if I run the command manually, apparently it's not even valid:

❯ go test -c -o d:\Work\odin\odin\api\src\services\deal\__debug_bin.exe -gcflags all=-N -l -v -race .
go: unknown flag -l cannot be used with -c

@thanm
Copy link
Contributor

thanm commented Aug 26, 2022

c:\program files\x86_64-w64-mingw32-native\include\syslimits.h:12:25: error: no include path in which to search for limits.h
12 | #include_next <limits.h>
| ^ (exit status 2)

I think this is getting a bit farther afield from the original issue ("ThreadSanitizer failed to allocate"). Seems like maybe something is out of whack with your gcc installation.

❯ go test -c -o d:\Work\odin\odin\api\src\services\deal__debug_bin.exe -gcflags all=-N -l -v -race .
go: unknown flag -l cannot be used with -c

That looks as though the "-l" is being interpreted by the Go command and not passed to the compiler. I would try fixing up your quoting, e.g. "-gcflags all=-N -l" etc.

@Southclaws
Copy link

Seems like maybe something is out of whack with your gcc installation.

I forgot this even used gcc haha I'm not sure which version I have installed, I always assumed Go on Windows used the native Microsoft compiler for C code.

I would try fixing up your quoting

none of these are commands I've written, these are just what comes out when I click "debug test" in vscode

I'll experiment a bit more next week, this was working fine for literal years and suddenly one day it all breaks and I can't debug any more, so annoying.

@benitogf
Copy link

Hello , I got this on 1.19

Setup go stable version spec 1.19.x
Found in cache @ C:\hostedtoolcache\windows\go\1.19.1\x64
Added go to the path
Successfully setup go version 1.19.x
go version go1.19.1 windows/amd64

go env
  set GO111MODULE=
  set GOARCH=amd64
  set GOBIN=
  set GOCACHE=C:\Users\runneradmin\AppData\Local\go-build
  set GOENV=C:\Users\runneradmin\AppData\Roaming\go\env
  set GOEXE=.exe
  set GOEXPERIMENT=
  set GOFLAGS=
  set GOHOSTARCH=amd64
  set GOHOSTOS=windows
  set GOINSECURE=
  set GOMODCACHE=C:\Users\runneradmin\go\pkg\mod
  set GONOPROXY=
  set GONOSUMDB=
  set GOOS=windows
  set GOPATH=C:\Users\runneradmin\go
  set GOPRIVATE=
  set GOPROXY=https://proxy.golang.org,direct
  set GOROOT=C:\hostedtoolcache\windows\go\1.19.1\x64
  set GOSUMDB=sum.golang.org
  set GOTMPDIR=
  set GOTOOLDIR=C:\hostedtoolcache\windows\go\1.19.1\x64\pkg\tool\windows_amd64
  set GOVCS=
  set GOVERSION=go1.19.1
  set GCCGO=gccgo
  set GOAMD64=v1
  set AR=ar
  set CC=gcc
  set CXX=g++
  set CGO_ENABLED=1
  set GOMOD=NUL
  set GOWORK=
  set CGO_CFLAGS=-g -O2
  set CGO_CPPFLAGS=
  set CGO_CXXFLAGS=-g -O2
  set CGO_FFLAGS=-g -O2
  set CGO_LDFLAGS=-g -O2
  set PKG_CONFIG=pkg-config
  set GOGCCFLAGS=-m64 -mthreads -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\RUNNER~1\AppData\Local\Temp\go-build372[8](https://github.com/benitogf/level/actions/runs/3118570570/jobs/5057948602#step:3:9)52[9](https://github.com/benitogf/level/actions/runs/3118570570/jobs/5057948602#step:3:10)3[11](https://github.com/benitogf/level/actions/runs/3118570570/jobs/5057948602#step:3:12)=/tmp/go-build -gno-record-gcc-switches
==1188==ERROR: ThreadSanitizer failed to allocate 0x000100000000 (4294967296) bytes at 0x040301000000 (error code: 1455)
exit status 66

@dvyukov
Copy link
Member

dvyukov commented Sep 24, 2022

@benitogf you are out of memory:

1455 | The paging file is too small for this operation to complete. | ERROR_COMMITMENT_LIMIT

@benitogf
Copy link

got it, thanks @dvyukov 😅

@alexbrainman
Copy link
Member

@benitogf you are out of memory:

@dvyukov would it be helpful to display Windows error message as well as error code in ThreadSanitizer error message? I suspect some users that get to see ThreadSanitizer error messages might not know how to lookup these Windows errors by their code.

Alex

@dvyukov
Copy link
Member

dvyukov commented Sep 25, 2022

Probably yes.

vvejell1 pushed a commit to vvejell1/containerd that referenced this issue Nov 4, 2022
This reverts commit 1ef4bda.

Previously we were downgrading mingw to work around an issue in the race
detector in Go on Windows when used with a newer version of GCC. The
issue was first reported here:

golang/go#46099

Shortly after the release of 1.19 someone had commented this issue was
solved for them, and after trying it out in some test runs on actions
machines, it seems to be the case. Disabling ASLR got things in order, and
PIE was disabled for -race builds in 1.19, so this is likely the reason
things work now:
golang/go@0c7fcf6.

The downgrade was mostly harmless except for two shortcomings:

1. It took quite a while for the package to get downloaded+installed.

2. Chocolatey would frequently fail to download with `The remote file
either doesn't exist, is unauthorized, or is forbidden for url ...
Exception calling "GetResponse" with "0" argument(s): "The request
was aborted: Could not create SSL/TLS secure channel."` Restarting the
failed run would often resolve this, but a 50-50 shot of things working
is not a great situation.

Signed-off-by: Daniel Canter <dcanter@microsoft.com>
n8maninger added a commit to SiaFoundation/renterd that referenced this issue Nov 9, 2022
n8maninger added a commit to SiaFoundation/renterd that referenced this issue Nov 9, 2022
peterjan pushed a commit to SiaFoundation/renterd that referenced this issue Nov 9, 2022
fuweid added a commit to fuweid/bbolt that referenced this issue Jan 3, 2023
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows
Projects
Status: Done
Development

No branches or pull requests