dev: running --race is very slow

[ajwerner]

Describe the problem

When you switch to running a test with --race, several bad things happen:

1. All of the generated code gets rebuilt
2. The protoc code generator gets rebuilt
3. The code generators are built with race, so they run super slowly

Jira issue: CRDB-15245

Epic CRDB-17171

[ajwerner]

This feels like maybe it's a blocker. I just tried to run a race test and it spent over 200s just working on the go stdlib before I gave up. That's way too long. I was able to do a ./dev generate go; make stressrace PKG=./pkg/sql/gcjob_test TESTS=... 2>&1 | tee out and it got to the test before the stdlib had compiled. To be clear, that was also miserable and slow, but bazel being so much slower for something not all too uncommon hurts.

[rickystewart]

Opened bazelbuild/rules_go#3218 for discussion.

[rickystewart]

The issue I filed doesn't cover it taking too long to build the stdlib, I'm not really sure what to do about that. Of course you should "only have to do it once" if caching is working as expected but still the 200s wait is a lot even just once.

[ajwerner]

That's the part I don't really understand. It seems to me that when you run with race in the normal go compiler it doesn't seem to have to rebuild the stdlib. Another issue is that, I think, it invalidates all of the generated code. I think this is a good example of thing where a little consulting might help.

[rickystewart]

It seems to me that when you run with race in the normal go compiler it doesn't seem to have to rebuild the stdlib.

When you build anything with the normal go compiler it doesn't have to build the stdlib, but that's just a difference between the normal Go toolchain and rules_go. Presumably the Go toolchain comes with a version of the stdlib pre-built with race detection.

[jfirebaugh]

Presumably the Go toolchain comes with a version of the stdlib pre-built with race detection.

It does, but all the precompiled artifacts are in the SDK are slated to be removed: golang/go#47257