Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/release: windows-amd64 release process is failing on tip due to unexpected stale targets after second build #52009

Closed
dmitshur opened this issue Mar 29, 2022 · 6 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Milestone

Comments

@dmitshur
Copy link
Contributor

As detected early by @heschi via work on #51797, Go 1.19 tip tree that will be used during the release process is not passing on the windows/amd64 release target.

Bisection shows the problem starts with CL 391809 (CC @bcmills). Either there's a problem in that CL that is uncovered only during the sequence of steps the current release process performs (since it's not being caught by builders), or there's a problem with that sequence which is only being uncovered by that change.

Its parent commit (d68615f) gets as far as API check successfully, but commit 0433f57 fails early in the "Building (all.bash to ensure tests pass)" step with:

# (The release command uses builders to create a release artifact locally.
#  It does not publish anything, so it's safe to run for testing needs.)
$ release -target=windows-amd64 -version=go1.19alpha126 -watch -rev=0433f5770b9c92b06d5a15c8d2a70f00597faac4
[...]
2022/03/28 18:59:19 windows-amd64: Installing go1.4 (second time, for all.bash).
2022/03/28 18:59:22 windows-amd64: Building (all.bash to ensure tests pass).
Building Go cmd/dist using C:\workdir\go1.4
Building Go toolchain1 using C:\workdir\go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for windows/amd64.

HASH[build internal/goarch]
[...]
HASH[link cmd/dist]: bdb2e2909c821c8e0277ce92f8c75b485d05999d0d32e4da1e549e1b9d4dbfab
cmd/dist true
go tool dist: unexpected stale targets reported by go list -gcflags="" -ldflags="" for [std cmd] (consider rerunning with GOMAXPROCS=1 GODEBUG=gocachehash=1):
	STALE cmd/addr2line: build ID mismatch
	STALE cmd/api: build ID mismatch
	STALE cmd/asm: build ID mismatch
	STALE cmd/buildid: build ID mismatch
	STALE cmd/cgo: build ID mismatch
	STALE cmd/compile: build ID mismatch
	STALE cmd/cover: build ID mismatch
	STALE cmd/dist: build ID mismatch
	STALE cmd/doc: build ID mismatch
	STALE cmd/fix: build ID mismatch
	STALE cmd/go: build ID mismatch
	STALE cmd/gofmt: build ID mismatch
	STALE cmd/internal/metadata: build ID mismatch
	STALE cmd/link: build ID mismatch
	STALE cmd/nm: build ID mismatch
	STALE cmd/objdump: build ID mismatch
	STALE cmd/pack: build ID mismatch
	STALE cmd/pprof: build ID mismatch
	STALE cmd/test2json: build ID mismatch
	STALE cmd/trace: build ID mismatch
	STALE cmd/vet: build ID mismatch

(complete log https://gist.github.com/dmitshur/80444d6afa37edb6dcdc2f2979545762)

This test failure can be reproduced with cmd/release (can only be done by someone with gomote access).

CC @golang/release.

@dmitshur dmitshur added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels Mar 29, 2022
@dmitshur dmitshur added this to the Go1.19 milestone Mar 29, 2022
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Mar 29, 2022
@bcmills bcmills self-assigned this Mar 30, 2022
@bcmills
Copy link
Member

bcmills commented Mar 31, 2022

As expected, running make.bat followed by all.bat on a raw builder does not reproduce this failure.

I suspect that it is something to do with the environment or args passed to the scripts. I'm adding some logging to cmd/release and re-running to try to suss that out.

@gopherbot
Copy link

Change https://go.dev/cl/398059 mentions this issue: cmd/go/internal/work: omit modinfo line from cache key when empty

@gopherbot
Copy link

Change https://go.dev/cl/398060 mentions this issue: run.bat: use cmd/dist instead of 'go install' to rebuild std and cmd

@gopherbot
Copy link

Change https://go.dev/cl/398058 mentions this issue: cmd/go: quote fragments in CGO_ env variables reported by 'go env'

@gopherbot
Copy link

Change https://go.dev/cl/398061 mentions this issue: cmd/dist: move more environment logic into cmd/dist from make and run scripts

@bcmills
Copy link
Member

bcmills commented Apr 5, 2022

I did eventually reproduce this by setting GO_BUILDER_NAME before running run.bat on a gomote instance. (cmd/dist's behavior changes based on whether it is running on a builder, because the builder is only supposed to run run.bat in a GOROOT that has already run make.bat and not been modified since.)

gopherbot pushed a commit that referenced this issue Apr 5, 2022
These fields have been parsed as quoted fields since CL 334732,
but we missed the unparsing side in 'go env'.

Certain scripts (notably make.ba{sh,t}) expect to be able to set the
environment to exactly what 'go env' reports, so for round-trip
purposes it is important to match the marshaling and unmarshaling
functions.

(Noticed while debugging #52009.)
Updates #41400

Change-Id: I0ff39b7a6e1328111c285c97cd23f79b723f3c73
Reviewed-on: https://go-review.googlesource.com/c/go/+/398058
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
gopherbot pushed a commit that referenced this issue Apr 5, 2022
Cache keys are dumped in case of mismatch; an empty modinfo string
adds noise to that dump without a corresponding benefit.

For #52009.

Change-Id: I1b4cd85fa5ff920973552fc94977954f93622a32
Reviewed-on: https://go-review.googlesource.com/c/go/+/398059
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
gopherbot pushed a commit that referenced this issue Apr 5, 2022
cmd/dist may set and/or unset variables before building, and at any
rate it is fragile to run 'go install' before sourcing env.bat.

The build-stamp information embedded by the 'go' command is currently
sensitive to whether CGO_* variables are implicit or explicit, so running
'go install' before env.bat may cause stamped metadata to become stale.
(Explicitly setting to the default arguably ought to produce the same
metadata as leaving the variables unset, but that's a separate issue
and a bigger cleanup.)

Moreover, run.bat is supposed to parallel run.bash, and run.bash
already hasn't invoked 'go install' itself since CL 6531!

For #52009

Change-Id: Ie35217913f02cc7e0c3f9b12874abd7416473478
Reviewed-on: https://go-review.googlesource.com/c/go/+/398060
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@rsc rsc unassigned bcmills Jun 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Projects
None yet
Development

No branches or pull requests

3 participants