New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

race: 'unsupported VMA range' on linux/arm64 #29948

Open
mengzhuo opened this Issue Jan 26, 2019 · 23 comments

Comments

Projects
None yet
7 participants
@mengzhuo
Copy link
Contributor

mengzhuo commented Jan 26, 2019

I think it might related to
https://github.com/llvm-mirror/compiler-rt/blob/master/lib/dfsan/dfsan.cc#L85

What version of Go are you using (go version)?

master

$ go version devel +8c10ce164f Fri Jan 25 15:16:56 2019 +0000 linux/arm64

Does this issue reproduce with the latest release?

No

go test -v -race
go test: -race is only supported on linux/amd64, linux/ppc64le, freebsd/amd64, netbsd/amd64, darwin/amd64 and windows/amd64

What operating system and processor architecture are you using (go env)?

Linux/arm64
CPU: RockChip 3399pro

go env Output
$ uname -a
Linux RK3399 4.4.167 #17 SMP Sat Jan 19 14:04:40 CST 2019 aarch64 aarch64 aarch64 GNU/Linux

$ go env
GOARCH="arm64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_arm64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build737658841=/tmp/go-build -gno-record-gcc-switches"

What did you do?

What did you expect to see?

ALL test passed.

What did you see instead?

## all non-race test passed

./all.bash -v
......
##### os/user with tag osusergo
ok      os/user 0.010s

##### GOMAXPROCS=2 runtime -cpu=1,2,4 -quick
ok      runtime 29.164s

##### cmd/go terminal test
PASS
ok      _/root/godev/src/cmd/go/testdata/testterminal18153      0.007s

##### sync -cpu=10
ok      sync    2.018s

##### Testing race detector
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    runtime/race    0.035s
2019/01/27 22:55:33 Failed: exit status 1
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    flag    0.041s
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    net     0.012s
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    os      0.015s
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    os/exec 0.009s
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    encoding/gob    0.012s
2019/01/27 22:55:37 Failed: exit status 1
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
exit status 66
FAIL    _/root/godev/misc/cgo/test      0.007s
2019/01/27 22:56:00 Failed: exit status 1
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    flag    0.013s
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    os/exec 0.016s
2019/01/27 22:56:01 Failed: exit status 1

##### ../misc/cgo/stdio
skipped due to earlier error

##### ../misc/cgo/life
skipped due to earlier error

##### ../misc/cgo/test
skipped due to earlier error

##### ../misc/cgo/testgodefs
skipped due to earlier error
2019/01/27 22:56:07 FAILED

@mengzhuo

This comment has been minimized.

Copy link
Contributor Author

mengzhuo commented Jan 26, 2019

#25682

It seems to be intensional not to support VMA 39 https://reviews.llvm.org/D52167

@mengzhuo

This comment has been minimized.

Copy link
Contributor Author

mengzhuo commented Jan 27, 2019

@Zheng-Xu

This comment has been minimized.

Copy link
Contributor

Zheng-Xu commented Jan 27, 2019

@mengzhuo Your observation is correct. It is a combined problem with Golang and compiler-rt. 39-bit VMA cannot easily be supported by TSAN for Go heap in compiler-rt. Actually there is also some problems with 42-bit VMA. TSAN runtime requires GO heap to be in the range [0x00c000000000, 0x00e000000000], a part of shadow memory region will be out of 42bits address(See: https://github.com/llvm-mirror/compiler-rt/blob/58d43607862096aeb32d72173911c9df244a30f1/lib/tsan/rtl/tsan_platform.h#L795). In theory, we should be able to support 47-bit VMA. But it is not a common configuration in Linux distos, so only 48-bit VMA is supported.

@cherrymui

This comment has been minimized.

Copy link
Contributor

cherrymui commented Jan 28, 2019

Thanks @mengzhuo and @Zheng-Xu for the investigation.

What is the right solution here? Reconfigure the kernel to use 48-bit VMA? Is there anything we can do on the Go side?

@Zheng-Xu

This comment has been minimized.

Copy link
Contributor

Zheng-Xu commented Jan 29, 2019

On the Go side, it is better that we ignore the tests on arm64 when the kernel is not configured as 48-bit VMA.

@cherrymui

This comment has been minimized.

Copy link
Contributor

cherrymui commented Jan 29, 2019

Thanks, @Zheng-Xu. How do we check that? Reading kernel config? Or we can just try to map some address and see if it works? Mind contributing a CL? Thanks.

We should probably also document this in the release note.

@Zheng-Xu

This comment has been minimized.

Copy link
Contributor

Zheng-Xu commented Feb 1, 2019

@cherrymui I took a quick look on disabling the test. But I don't have a good solution so far. I try to list what I found and the difficulty here:

Do you think above solution is reasonable? I can push a CL if it is OK.

@cherrymui

This comment has been minimized.

Copy link
Contributor

cherrymui commented Feb 1, 2019

Thanks for the information.

As you said, we should not do anything at build time, exactly because of cross compilation.

Always disabling -race test in cmd/dist is also not a good idea, as we'll not know if it breaks.

Maybe in cmd/dist, we can check the stderr and if it is the unsupported VMA failure, we turn it into SKIP instead of FAIL?

Users who use the race detector can still encounter this in their own programs. So we still need to document this.

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 3, 2019

Change https://golang.org/cl/160919 mentions this issue: cmd/dist: skip race detector test failure for unsupported VMA

@zorrorffm

This comment has been minimized.

Copy link
Contributor

zorrorffm commented Feb 3, 2019

@cherrymui Could you mind take a look the patch catching unsupported VMA failure and skipping that.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Feb 4, 2019

If I understand this correctly, the race detector simply doesn't work reliably on arm64 systems. And the problem can only be fixed in race.syso, not in Go code.

That's too bad, but I think that for 1.12 we should remove arm64 from raceDetectorSupported in cmd/dist/test.go and cmd/internal/sys/supported.go. It wasn't supported in 1.11, so this is not a regression. We can try again to enable it in 1.13.

@zorrorffm

This comment has been minimized.

Copy link
Contributor

zorrorffm commented Feb 13, 2019

@ianlancetaylor Actually for 44-bit VMA , ppc64le has same problem. Only 46-bit VMA and 47-bit VMA are currently supported for powercp64 on Go.

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 13, 2019

@ianlancetaylor, what do you want to do here (if anything) for Go 1.12? I see that https://go-review.googlesource.com/c/go/+/160919/ stalled and hasn't been updated with review feedback.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Feb 13, 2019

@cherrymui @mengzhuo Should we keep or disable arm64 race detector for the 1.12 release? I don't fully understand the circumstances under which it fails.

@mengzhuo

This comment has been minimized.

Copy link
Contributor Author

mengzhuo commented Feb 14, 2019

@ianlancetaylor I think keep linux/arm64 race detector is fine. This issue only happens when you try to run race builds on unsupported kernel

@mengzhuo mengzhuo closed this Feb 14, 2019

@mengzhuo mengzhuo reopened this Feb 14, 2019

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Feb 14, 2019

OK, thanks.

@cherrymui

This comment has been minimized.

Copy link
Contributor

cherrymui commented Feb 14, 2019

If I understand correctly, the ARM64 race detector support works fine under certain kernel configurations (probably the popular one?), but doesn't work at all under some other kernel configurations. So I was suggesting we skip the tests under unsupported kernel.

@bradfitz bradfitz modified the milestones: Go1.12, Go1.13 Feb 14, 2019

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 14, 2019

@cherrymui, skipping tests is fine (even for Go 1.12, or a later point release) if you want. But I don't think this is a release blocker at this point. That is, all.bash passing for everybody's distro/version isn't a blocker for shipping a release.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Feb 14, 2019

I agree that this is not a release-blocker but more generally I don't think skipping the tests is really a fix here. If we can skip the tests, we can and should also report the problem when running a binary built with -race.

@zorrorffm

This comment has been minimized.

Copy link
Contributor

zorrorffm commented Feb 14, 2019

@cherrymui @ianlancetaylor Currently data race works fine on the kernel configured with 48-bits VMA which is popular. Data race doesn't support the kernel configured with 39-bit VMA and 42-bit VMA because the shadow memory calculated in TSAN runtime will be out of the address space. I will investigate how to support 39-bit VMA in TSAN runtime.

@cherrymui

This comment has been minimized.

Copy link
Contributor

cherrymui commented Feb 14, 2019

@bradfitz I also agree that this does not block release.

@zorrorffm

This comment has been minimized.

Copy link
Contributor

zorrorffm commented Feb 14, 2019

@ianlancetaylor TSAN runtime will report the error message when running a binary built with -race on unsupported VMA.
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Feb 14, 2019

OK, thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment