x/build/cmd/coordinator: main Go tests don't pass when CGO_CFLAGS=-mmacosx-version-min=10.13 set in environment

[dmitshur]

CL 382055 set CGO_CFLAGS=-mmacosx-version-min=10.13 in the environment of the darwin-amd64-10_15 builder. With this change, all golang.org/x module tests continued to pass (e.g., x/arch build log), but the main Go tests started to fail with the following output:

darwin-amd64-10_15 at 896df422a7cecbace10f5877beeeb1476b6061ae

go tool dist: unexpected stale targets reported by go list -gcflags="" -ldflags="" for [std cmd] (consider rerunning with GOMAXPROCS=1 GODEBUG=gocachehash=1):
	STALE archive/tar: stale dependency: runtime/cgo
	STALE crypto/tls: stale dependency: runtime/cgo
	STALE crypto/x509: stale dependency: runtime/cgo
[...]
	STALE cmd/vendor/github.com/google/pprof/internal/symbolz: stale dependency: runtime/cgo
	STALE cmd/vendor/github.com/google/pprof/internal/transport: stale dependency: runtime/cgo
	STALE cmd/vendor/golang.org/x/mod/sumdb: stale dependency: runtime/cgo

(Source: https://build.golang.org/log/4474c782b89f18405361027bebcab80e09129190.)

Another test run, with more information captured, suggests the problem happens in the "run_tests_multi" step when the sharded tests start running:

  builder: darwin-amd64-10_15
      rev: 896df422a7cecbace10f5877beeeb1476b6061ae
 buildlet: http://macstadium_host03a reverse peer macstadium_host03a/207.254.3.10:56081 for host type host-darwin-10_15
  started: 2022-02-03 21:55:43.764155923 +0000 UTC m=+2657.016357431
    ended: 2022-02-03 22:03:37.386828895 +0000 UTC m=+3130.639030413
  success: false

Events:
  2022-02-03T21:55:43Z checking_for_snapshot 
  2022-02-03T21:55:43Z finish_checking_for_snapshot after 25.3ms
  2022-02-03T21:55:43Z get_buildlet 
  2022-02-03T22:03:26Z finish_get_buildlet after 7m42.1s
  2022-02-03T22:03:26Z using_buildlet macstadium_host03a
  2022-02-03T22:03:26Z write_snapshot_tar 
  2022-02-03T22:03:34Z finish_write_snapshot_tar after 7.6s
  2022-02-03T22:03:34Z make_and_test 
  2022-02-03T22:03:34Z discovering_tests 
  2022-02-03T22:03:35Z get_test_stats 
  2022-02-03T22:03:35Z finish_get_test_stats after 0s
  2022-02-03T22:03:35Z starting_tests 270 tests
  2022-02-03T22:03:35Z run_tests_multi macstadium_host03a: [go_test:archive/tar go_test:archive/zip go_test:bufio]
  2022-02-03T22:03:37Z finish_run_tests_multi after 1.31s; macstadium_host03a: [go_test:archive/tar go_test:archive/zip go_test:bufio]
  2022-02-03T22:03:37Z run_tests_multi macstadium_host03a: [go_test:bytes go_test:compress/bzip2 go_test:compress/flate]
  2022-02-03T22:03:37Z finish_make_and_test after 2.94s
  2022-02-03T22:03:37Z done with test failures
  2022-02-03T22:03:37Z finish_run_tests_multi after 209.7ms; err=buildlet: Client closed; macstadium_host03a: [go_test:bytes go_test:compress/bzip2 go_test:compress/flate]
  2022-02-03T22:03:37Z main_buildlet_broken macstadium_host03a

Build log:
darwin-amd64-10_15 at 896df422a7cecbace10f5877beeeb1476b6061ae

go tool dist: unexpected stale targets reported by go list -gcflags="" -ldflags="" for [std cmd] (consider rerunning with GOMAXPROCS=1 GODEBUG=gocachehash=1):
	STALE archive/tar: stale dependency: runtime/cgo
	STALE crypto/tls: stale dependency: runtime/cgo
	STALE crypto/x509: stale dependency: runtime/cgo
[...]
	STALE cmd/vendor/github.com/google/pprof/internal/symbolz: stale dependency: runtime/cgo
	STALE cmd/vendor/github.com/google/pprof/internal/transport: stale dependency: runtime/cgo
	STALE cmd/vendor/golang.org/x/mod/sumdb: stale dependency: runtime/cgo



Error: tests failed: dist test failed: go_test:archive/tar: exit status 1

CC @golang/release, @bcmills.

[dmitshur]

Writing down that when this was discussed previously, there was a hypothesis that this might've a false positive caused by a race involving the old snapshot of a built Go toolchain (built before the coordinator was redeployed with the CGO_CFLAGS env var in the builder configuration) being reused in the new build, since the snapshot is keyed only on the builder name and Go commit hash.

If that turns out to be it, we can consider improving snapshot key to include some a hash of the builder configuration or coordinator version.