runtime/race: race when building latest llvm tsan with go1.25 pre-release

[glycerine]

Go version

go version go1.25-prerelease-b062eb46e8 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE='auto'
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/jaten/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/jaten/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2866629629=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD=''
GOMODCACHE='/home/jaten/go/pkg/mod'
GONOPROXY='github.com/glycerine/*'
GONOSUMDB='github.com/glycerine/*'
GOOS='linux'
GOPATH='/home/jaten/go'
GOPRIVATE='github.com/glycerine/*'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/mnt/oldrog/usr/local/dev-go/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='off'
GOTELEMETRYDIR='/home/jaten/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/mnt/oldrog/usr/local/dev-go/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25-prerelease-b062eb46e8'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

I was trying to characterize the race detector/tsan bugs found in

#74019 . While originally found in go1.24.3, these
issues have persisted into go1.25 pre-release (b062eb4).

See also https://github.com/glycerine/rr_binary_for_issue74019?tab=readme-ov-file#checking-on-go125-prerelease-at-b062eb46e8).

So I tried to build the latest tsan thread-sanitizer from the llvm tree. The thought was,
perhaps there have been fixes in the last 20 months to tsan. If bugs have already
been fixed in tsan, then maybe that will help with #74019.

https://github.com/golang/go/blob/master/src/runtime/race/README says that
the linux amd64 .syso is from

internal/amd64v1/race_linux.syso built with LLVM 51bfeff0e4b0757ff773da6882f4d538996c9b04 and Go e7d582b55dda36e76ce4d0ce770139ca0915b7c5.

which is from llvm/llvm-project@51bfeff -- committed 2023 Oct 20; quite a while back.

I executed the same commands that "racebuild" build does in the embedded shell script, since
my attempt to run racebuild itself, per the instructions in src/runtime/race/README did not work (it uses gomote, and that needs a special gmail account maybe--I signed in to gmail but do not have permissions);
thus the identical commands, as read from https://cs.opensource.google/go/x/build/+/master:cmd/racebuild/racebuild.go for
the linux amd64 target, are done manually below.

The attempt to build the race detector .syso for linux amd64 failed with a race itself.

Ignoring the massive irony, fixing the build of tsan could be a good starting point for addressing the
#74019 bugs.

jaten@rog /usr $ cd local/dev-go/go/
jaten@rog /usr/local/dev-go/go (go1.25-goj) $ ls
api             CONTRIBUTING.md  lib      PATENTS    SECURITY.md  VERSION
bin             doc              LICENSE  pkg        src          VERSION~
codereview.cfg  go.env           misc     README.md  test
jaten@rog /usr/local/dev-go/go (go1.25-goj) $ cd ..
jaten@rog /usr/local/dev-go $ mkdir tmp
jaten@rog /usr/local/dev-go $ cd tmp
jaten@rog /usr/local/dev-go/tmp $ curl -L -o llvm.zip https://github.com/llvm/llvm-project/archive/46e3ec0244c4d75a57cd635a28a9d5cbaee67c3d.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  318M    0  318M    0     0  6697k      0 --:--:--  0:00:48 --:--:-- 6865k
jaten@rog /usr/local/dev-go/tmp $ unzip -q llvm.zip llvm-project-46e3ec0244c4d75a57cd635a28a9d5cbaee67c3d/compiler-rt/* 
jaten@rog /usr/local/dev-go/tmp $ ls
llvm-project-46e3ec0244c4d75a57cd635a28a9d5cbaee67c3d  llvm.zip
jaten@rog /usr/local/dev-go/tmp $ cd llvm-project-46e3ec0244c4d75a57cd635a28a9d5cbaee67c3d/
jaten@rog /usr/local/dev-go/tmp/llvm-project-46e3ec0244c4d75a57cd635a28a9d5cbaee67c3d $ cd compiler-rt/lib/tsan/go
jaten@rog /usr/local/dev-go/tmp/llvm-project-46e3ec0244c4d75a57cd635a28a9d5cbaee67c3d/compiler-rt/lib/tsan/go $ ls
build.bat  buildgo.sh  test.c  tsan_go.cpp
jaten@rog /usr/local/dev-go/tmp/llvm-project-46e3ec0244c4d75a57cd635a28a9d5cbaee67c3d/compiler-rt/lib/tsan/go $ ./buildgo.sh 
In file included from ../rtl/tsan_sync.h:19,
                 from ../rtl/tsan_rtl.h:47,
                 from tsan_go.cpp:13:
../rtl/tsan_dense_alloc.h: In instantiation of ‘void __tsan::DenseSlabAlloc< <template-parameter-1-1>, <anonymous>, <anonymous>, <anonymous> >::Drain(Cache*) [with T = __tsan::MBlock; long unsigned int kL1Size = 262144; long unsigned int kL2Size = 4096; long long unsigned int kReserved = 3221225472; Cache = __tsan::DenseSlabAllocCache]’:
../rtl/tsan_dense_alloc.h:77:7:   required from ‘void __tsan::DenseSlabAlloc< <template-parameter-1-1>, <anonymous>, <anonymous>, <anonymous> >::Free(Cache*, IndexT) [with T = __tsan::MBlock; long unsigned int kL1Size = 262144; long unsigned int kL2Size = 4096; long long unsigned int kReserved = 3221225472; Cache = __tsan::DenseSlabAllocCache; IndexT = unsigned int]’
../rtl/tsan_sync.cpp:84:26:   required from here
../rtl/tsan_dense_alloc.h:165:46: warning: suggest parentheses around arithmetic in operand of ‘|’ [�]8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wparentheses�-Wparentheses�]8;;�]
  165 |       xchg = head_idx | (cmp & kCounterMask) + kCounterInc;
      |                         ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~
../rtl/tsan_dense_alloc.h: In instantiation of ‘void __tsan::DenseSlabAlloc< <template-parameter-1-1>, <anonymous>, <anonymous>, <anonymous> >::Drain(Cache*) [with T = __tsan::SyncVar; long unsigned int kL1Size = 1048576; long unsigned int kL2Size = 1024; long long unsigned int kReserved = 3221225472; Cache = __tsan::DenseSlabAllocCache]’:
../rtl/tsan_dense_alloc.h:77:7:   required from ‘void __tsan::DenseSlabAlloc< <template-parameter-1-1>, <anonymous>, <anonymous>, <anonymous> >::Free(Cache*, IndexT) [with T = __tsan::SyncVar; long unsigned int kL1Size = 1048576; long unsigned int kL2Size = 1024; long long unsigned int kReserved = 3221225472; Cache = __tsan::DenseSlabAllocCache; IndexT = unsigned int]’
../rtl/tsan_sync.cpp:92:25:   required from here
../rtl/tsan_dense_alloc.h:165:46: warning: suggest parentheses around arithmetic in operand of ‘|’ [�]8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wparentheses�-Wparentheses�]8;;�]
cc1plus: note: unrecognized command-line option ‘-Wno-unknown-warning-option’ may have been intended to silence earlier diagnostics
gcc gotsan.cpp -c -o ./race_linux_amd64.syso -I../rtl -I../.. -I../../sanitizer_common -I../../../include -std=c++17 -Wall -fno-exceptions -fno-rtti -DSANITIZER_GO=1 -DSANITIZER_DEADLOCK_DETECTOR_VERSION=2 -fPIC -Wno-maybe-uninitialized -ffreestanding -Wno-unused-const-variable -Wno-unknown-warning-option -m64 -msse3 -DSANITIZER_DEBUG=0 -O3 -fomit-frame-pointer
In file included from ../rtl/tsan_sync.h:19,
                 from ../rtl/tsan_rtl.h:47,
                 from tsan_go.cpp:13:
../rtl/tsan_dense_alloc.h: In instantiation of ‘void __tsan::DenseSlabAlloc< <template-parameter-1-1>, <anonymous>, <anonymous>, <anonymous> >::Drain(Cache*) [with T = __tsan::MBlock; long unsigned int kL1Size = 262144; long unsigned int kL2Size = 4096; long long unsigned int kReserved = 3221225472; Cache = __tsan::DenseSlabAllocCache]’:
../rtl/tsan_dense_alloc.h:77:7:   required from ‘void __tsan::DenseSlabAlloc< <template-parameter-1-1>, <anonymous>, <anonymous>, <anonymous> >::Free(Cache*, IndexT) [with T = __tsan::MBlock; long unsigned int kL1Size = 262144; long unsigned int kL2Size = 4096; long long unsigned int kReserved = 3221225472; Cache = __tsan::DenseSlabAllocCache; IndexT = unsigned int]’
../rtl/tsan_sync.cpp:84:26:   required from here
../rtl/tsan_dense_alloc.h:165:46: warning: suggest parentheses around arithmetic in operand of ‘|’ [�]8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wparentheses�-Wparentheses�]8;;�]
  165 |       xchg = head_idx | (cmp & kCounterMask) + kCounterInc;
      |                         ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~
../rtl/tsan_dense_alloc.h: In instantiation of ‘void __tsan::DenseSlabAlloc< <template-parameter-1-1>, <anonymous>, <anonymous>, <anonymous> >::Drain(Cache*) [with T = __tsan::SyncVar; long unsigned int kL1Size = 1048576; long unsigned int kL2Size = 1024; long long unsigned int kReserved = 3221225472; Cache = __tsan::DenseSlabAllocCache]’:
../rtl/tsan_dense_alloc.h:77:7:   required from ‘void __tsan::DenseSlabAlloc< <template-parameter-1-1>, <anonymous>, <anonymous>, <anonymous> >::Free(Cache*, IndexT) [with T = __tsan::SyncVar; long unsigned int kL1Size = 1048576; long unsigned int kL2Size = 1024; long long unsigned int kReserved = 3221225472; Cache = __tsan::DenseSlabAllocCache; IndexT = unsigned int]’
../rtl/tsan_sync.cpp:92:25:   required from here
../rtl/tsan_dense_alloc.h:165:46: warning: suggest parentheses around arithmetic in operand of ‘|’ [�]8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wparentheses�-Wparentheses�]8;;�]
cc1plus: note: unrecognized command-line option ‘-Wno-unknown-warning-option’ may have been intended to silence earlier diagnostics
==================
WARNING: DATA RACE
Read at 0x00c011110000 by goroutine 2:
  <null>()
      <null>:0 +0x0

Previous write at 0x00c011110000 by main goroutine:
  <null>()
      <null>:0 +0x0
  <null>()
      <null>:0 +0x0

Goroutine 2 (running) created at:
  <null>()
      <null>:0 +0x0
==================
Found 1 data race(s)
jaten@rog /usr/local/dev-go/tmp/llvm-project-46e3ec0244c4d75a57cd635a28a9d5cbaee67c3d/compiler-rt/lib/tsan/go $

What did you see happen?

see above.

What did you expect to see?

Ability to re-build the race detector with the latest from the llvm tree, so I could
evaluate whether that helps with the #74019 bugs.

Originally queried in https://groups.google.com/g/golang-nuts/c/Tuo829iE0q0

[gabyhelp]

Related Issues

• runtime/race: llvm-unstable changes break Go tests with -race: panic: release of handle with refcount 0 #70283 (closed)
• cmd/go: missing __tsan_* functions in cross-compiled windows/amd64 race build #42382 (closed)
• Crosscompile with race detector fails with fault and panic #49102 (closed)
• gccgo: TestShared/tsan_shared fails with GCC 5.5.0 #24046
• race: Thread Sanitizer breaks sequential consistency semantics #58020 (closed)
• go/cmd go race checker: runtime.(*mspan).setUserArenaChunkToFault relocation target not defined #60624 (closed)

Related Code Changes

• runtime/race: implement race detector for ppc64le
• runtime/race: update race_windows_amd64.syso
• runtime/race: update syso files to new TSAN runtime (v3)
• cmd,runtime: enable race detector on loong64

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[glycerine]

Ah. I see now what happened. The build script runs a test after the build is finished, to check that a race is detected. I do see linux .syso files being created, so what I took to be a build failure was actually a successful test of the newly built race detector library.

I'm almost ready to close this out as it actually does seem to be building. I'll try the new .syso next.

One last question though:

why does the race detector in use by the latest Go release lag 20 months behind that in llvm?

@dvyukov is there some concern about compatibility or something where we don't expect that latest tsan in llvm tree to be compatible with the latest Go?

Thanks!

[randall77]

It is just that it is work to build a new race detector for all the platforms, so we generally don't do it unless there is a race detector feature/bugfix that we want.

[glycerine]

@randall77 Thanks Keith. So the expectation is that the latest tsan in llvm should work, right?

I'm seeing some different errors with the latest tsan under the #74019 reproducers, which may be useful, especially as some of them give new/(maybe) more useful error messages like

ThreadSanitizer: CHECK failed: tsan_rtl_thread.cpp:261 "((parts->Queued(trace.local_head))) != (0)" (0x0, 0x0) (tid=342934)

[randall77]

@randall77 Thanks Keith. So the expectation is that the latest tsan in llvm should work, right?

That is the expectation. In the past I've always had a bit of friction pulling in a new tsan version. Something doesn't compile on our builders, some assumption about our runtime no longer holds, etc. But generally it should work.

[glycerine]

Thanks Keith. I skimmed through the 200 git log changes since 2023-Oct-20 to the files the race detector uses.

I'm no expert but nothing really stood out as a critical fix.

There are some things for Go related to #64345; riscv support related, but it doesn't look like that issue is ready to ship anyway--once it does they will need 76910f914cdd4b86b28e0d5852155244ee47dc53 from Feb 28 18:27:48 2025.

So I don't see any (glaring) cause to upgrade either at this point. I just prefer to not waste time just in case there is something subtle. I'll close this issue since its really just due to my not groking the race detector build script. Thanks for your advice.