Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: "unexpected stale targets" on darwin builders #33598

Open
bcmills opened this issue Aug 12, 2019 · 19 comments
Open

x/build: "unexpected stale targets" on darwin builders #33598

bcmills opened this issue Aug 12, 2019 · 19 comments

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Aug 12, 2019

Two flakes in the darwin-amd64-10_11 builders over the past week or so, both with the same symptom: unexpected stale targets failures for internal/cpu, and a build ID mismatch for runtime/internal/sys.

CC @jayconrod @ianlancetaylor @aclements

darwin-amd64-10_11 at 0ca4f6be35d2abf54128b57c3aa1a1fcbd5adfe3

:: Running /var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/make.bash with args ["/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/make.bash"] and env ["TERM_PROGRAM=Apple_Terminal" "TERM=xterm-256color" "SHELL=/bin/bash" "TMPDIR=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/" "Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.55fT0Y9EVR/Render" "TERM_PROGRAM_VERSION=361.1" "TERM_SESSION_ID=D33F650A-CE77-4BF0-B95E-A37406DCC122" "USER=gopher" "SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.43nQd98RPC/Listeners" "__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0" "PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin" "PWD=/Users/gopher" "LANG=en_US.UTF-8" "XPC_FLAGS=0x0" "XPC_SERVICE_NAME=0" "SHLVL=2" "HOME=/Users/gopher" "LOGNAME=gopher" "SECURITYSESSIONID=186a8" "GO_BUILDER_ENV=macstadium_vm" "GO_STAGE0_NET_DELAY=300ms" "GO_STAGE0_DL_DELAY=0s" "WORKDIR=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir" "GOROOT_BOOTSTRAP=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go1.4" "GO_BUILDER_NAME=darwin-amd64-10_11" "GOROOT_BOOTSTRAP=/Users/gopher/go1.4" "GOBIN=" "TMPDIR=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/tmp" "GOCACHE=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/gocache"] in dir /var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src

Building Go cmd/dist using /Users/gopher/go1.4.
Building Go toolchain1 using /Users/gopher/go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
HASH[build runtime/internal/sys]
HASH[build runtime/internal/sys]: "devel 0ca4f6be35d2abf54128b57c3aa1a1fcbd5adfe3"
HASH[build runtime/internal/sys]: "compile\n"
HASH[build runtime/internal/sys]: "goos darwin goarch amd64\n"
HASH[build runtime/internal/sys]: "import \"runtime/internal/sys\"\n"
HASH[build runtime/internal/sys]: "omitdebug false standard true local false prefix \"\"\n"
HASH[build runtime/internal/sys]: "modinfo \"\"\n"
HASH[build runtime/internal/sys]: "compile PRWJHAzAczrrtWSoyeY4 [] []\n"
HASH[build runtime/internal/sys]: "=\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/arch.go: d9b0b7e72538d421b2607acaba60ca49f20ef584b3d1d191c6729e35fbb8101d
HASH[build runtime/internal/sys]: "file arch.go 2bC35yU41CGyYHrKumDK\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/arch_amd64.go: d301f8642ec7fcc35d0d39a4b07e0187971afea7c6d0dfd5ddcef1d031670160
HASH[build runtime/internal/sys]: "file arch_amd64.go 0wH4ZC7H_MNdDTmksH4B\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/intrinsics.go: 8b469a461e1d983706e0b3635715ce70691adc5db7c4e067b88cc59f40cd66f4
HASH[build runtime/internal/sys]: "file intrinsics.go i0aaRh4dmDcG4LNjVxXO\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/stubs.go: 23b3e5c631b086fe7a2dec4bf044600e034bf6a8eeb25e0a19efc4ce6311423d
HASH[build runtime/internal/sys]: "file stubs.go I7PlxjGwhv56LexL8ERg\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/sys.go: 55e021891200a7e6a5c371c8a1ab71b6c15aeb16ea6c1b192185d17df8c8b18f
HASH[build runtime/internal/sys]: "file sys.go VeAhiRIAp-alw3HIoatx\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/zgoarch_amd64.go: 6f2fa4ebe8999a3b7cfe3ead6e24b8356ed842292f23cb1b4f995c0b5b45126b
HASH[build runtime/internal/sys]: "file zgoarch_amd64.go by-k6-iZmjt8_j6tbiS4\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/zgoos_darwin.go: 464151a4ab63f8c7aab345f457789cb12c9b68051fc73b8a675717d7886214dc
HASH[build runtime/internal/sys]: "file zgoos_darwin.go RkFRpKtj-Meqs0X0V3ic\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/zversion.go: 623491756e2adacebf4c53078883e4284349c6159d0404aba9975887d7660d06
HASH[build runtime/internal/sys]: "file zversion.go YjSRdW4q2s6_TFMHiIPk\n"
HASH[build runtime/internal/sys]: cd175367cc0e62c53ed8e20197f8e44b2270f61d3eb7b4f3c1991a5abb06ae16
runtime/internal/sys true
go tool dist: unexpected stale targets reported by /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/pkg/tool/darwin_amd64/go_bootstrap list -gcflags="" -ldflags="" for [cmd/asm cmd/cgo cmd/compile cmd/link runtime/internal/sys]:
	STALE cmd/asm: stale dependency: internal/cpu
	STALE cmd/cgo: stale dependency: internal/cpu
	STALE cmd/compile: stale dependency: internal/cpu
	STALE cmd/link: stale dependency: internal/cpu
	STALE runtime/internal/sys: build ID mismatch
@bcmills bcmills added this to the Go1.14 milestone Aug 12, 2019
@bcmills bcmills changed the title cmd/dist: "unexpected stale targets" for internal/cpu on darwin-amd64-10_11 builder cmd/dist: "unexpected stale targets" during toolchain3 bootstrapping on darwin-amd64-10_11 builder Aug 12, 2019
@bcmills
Copy link
Member Author

@bcmills bcmills commented Aug 27, 2019

@bcmills bcmills changed the title cmd/dist: "unexpected stale targets" during toolchain3 bootstrapping on darwin-amd64-10_11 builder cmd/dist: "unexpected stale targets" during toolchain3 bootstrapping on darwin builders Aug 27, 2019
@bcmills
Copy link
Member Author

@bcmills bcmills commented Sep 11, 2019

@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@bcmills
Copy link
Member Author

@bcmills bcmills commented Oct 22, 2019

Not clear to me whether this is during toolchain3, or if the underlying cause is even related, but there is a similar symptom on the linux-ppc64le-power9osu builder in https://build.golang.org/log/57277f725843345df871d34cfabf1fda58934be0.

@laboger
Copy link
Contributor

@laboger laboger commented Oct 22, 2019

I see the same type of failure with the stale targets when I try to force -pie to be the default build mode for ppc64le (due to a request). It is noted in #28531. Maybe something wrong with the way that internal/cpu is hashed so that it doesn't get consistent results?

@bcmills
Copy link
Member Author

@bcmills bcmills commented Nov 21, 2019

2019-11-11T17:53:51-70be481/darwin-amd64-10_11
2019-11-07T18:39:03-05aa4a7/darwin-amd64-10_11
2019-11-07T08:25:32-7a2baa9/darwin-amd64-10_11
2019-10-31T01:09:14-851b1f4/illumos-amd64
2019-10-31T01:09:14-851b1f4/solaris-amd64-oraclerel
2019-10-31T01:04:59-e90e6e7/illumos-amd64
2019-10-31T01:04:59-e90e6e7/solaris-amd64-oraclerel
2019-10-25T17:41:36-d77b809/illumos-amd64
2019-10-25T17:41:36-d77b809/solaris-amd64-oraclerel
2019-10-25T17:15:50-2e2ef66/illumos-amd64
2019-10-25T17:15:50-2e2ef66/solaris-amd64-oraclerel
2019-10-25T16:51:18-376ef73/illumos-amd64
2019-10-25T16:51:18-376ef73/solaris-amd64-oraclerel
2019-10-24T15:24:20-fb06609/illumos-amd64
2019-10-24T15:24:20-fb06609/solaris-amd64-oraclerel
2019-10-24T15:22:31-df01b79/illumos-amd64
2019-10-24T15:22:31-df01b79/solaris-amd64-oraclerel
2019-10-24T00:53:39-4adf822/illumos-amd64
2019-10-24T00:53:39-4adf822/solaris-amd64-oraclerel
2019-10-21T21:58:03-c480d32/illumos-amd64
2019-10-21T21:58:03-c480d32/solaris-amd64-oraclerel
2019-10-21T21:57:32-d56c814/illumos-amd64
2019-10-21T21:57:32-d56c814/solaris-amd64-oraclerel
2019-10-21T21:57:10-4fa524b/illumos-amd64
2019-10-21T21:57:10-4fa524b/solaris-amd64-oraclerel
2019-10-21T21:57:01-97e497b/illumos-amd64
2019-10-19T15:18:32-5777ffd/solaris-amd64-oraclerel
2019-10-18T18:44:05-c3459ea/solaris-amd64-oraclerel
2019-10-17T20:28:28-bd22993/solaris-amd64-oraclerel
2019-10-17T20:28:15-286d744/illumos-amd64
2019-10-17T20:28:15-286d744/solaris-amd64-oraclerel
2019-10-17T20:08:33-b661547/illumos-amd64
2019-10-17T20:08:33-b661547/solaris-amd64-oraclerel
2019-10-17T14:20:08-e5acb58/solaris-amd64-oraclerel
2019-10-17T14:20:00-6cbf37b/solaris-amd64-oraclerel
2019-10-17T14:18:37-8011051/solaris-amd64-oraclerel
2019-10-17T14:18:28-15634a0/solaris-amd64-oraclerel
2019-10-16T22:28:18-2bbf2e0/solaris-amd64-oraclerel
2019-10-16T15:57:07-5caac2f/solaris-amd64-oraclerel
2019-10-16T13:24:55-c9470b0/solaris-amd64-oraclerel
2019-10-15T19:00:07-4a9b30c/solaris-amd64-oraclerel
2019-10-15T19:00:00-7256f50/solaris-amd64-oraclerel
2019-10-15T18:59:52-5fec788/solaris-amd64-oraclerel
2019-10-15T18:56:08-6ba3ae9/solaris-amd64-oraclerel
2019-10-15T18:55:59-dab05a0/solaris-amd64-oraclerel
2019-10-15T18:55:48-27111e5/solaris-amd64-oraclerel
2019-10-15T18:55:32-cab29eb/solaris-amd64-oraclerel
2019-09-11T12:19:26-11c2411/darwin-amd64-10_14
2019-08-31T00:47:27-e5e5fb0/darwin-amd64-10_11
2019-08-26T19:39:05-97edf77/darwin-amd64-nocgo
2019-08-08T16:16:15-f4be93a/darwin-amd64-10_11
2019-08-07T14:40:17-0ca4f6b/darwin-amd64-10_11
2019-07-01T21:30:23-d410642/darwin-amd64-10_12

@bcmills bcmills changed the title cmd/dist: "unexpected stale targets" during toolchain3 bootstrapping on darwin builders cmd/dist: "unexpected stale targets" during toolchain3 bootstrapping on darwin and solaris builders Nov 21, 2019
@bcmills
Copy link
Member Author

@bcmills bcmills commented Nov 21, 2019

@laboger, the ppc64le failures seem to occur at a different point in the build process. I'll file a separate issue for that.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Feb 20, 2020

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 18, 2020

Curiously, the only builder that has failed in this way in the last month or so is the plan9-arm builder.

(There hasn't been another darwin-amd64 failure since February, and we have a bit of circumstantial evidence from linker crashes that that rate of failure may be due to platform filesystem instability.)

2020-03-02T17:40:11-a2bff7c/plan9-arm
2020-02-19T21:26:10-7299308/darwin-amd64-10_12
2020-02-18T19:44:25-a0cf2c8/plan9-arm

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 18, 2020

Curiously, CL 223755 seems to demonstrate that the darwin-amd64-10_12 builder is receiving a stale snapshot (or not using snapshots and not rebuilding appropriately).

Building Go cmd/dist using /Users/gopher/go1.4. (go1.4-bootstrap-20161024 darwin/amd64)
Building Go toolchain1 using /Users/gopher/go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.
---
Installed Go for darwin/amd64 in /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go
Installed commands in /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/bin
…
go tool dist: unexpected stale targets reported by go list -gcflags="" -ldflags="" for [std cmd]:
	STALE cmd/addr2line: build ID mismatch
	STALE cmd/api: build ID mismatch
	STALE cmd/asm: build ID mismatch
	STALE cmd/buildid: build ID mismatch
	STALE cmd/cgo: build ID mismatch
	STALE cmd/compile: build ID mismatch
	STALE cmd/cover: build ID mismatch
	STALE cmd/dist: build ID mismatch
	STALE cmd/doc: build ID mismatch
	STALE cmd/fix: build ID mismatch
	STALE cmd/go: build ID mismatch
	STALE cmd/gofmt: build ID mismatch
	STALE cmd/link: build ID mismatch
	STALE cmd/nm: build ID mismatch
	STALE cmd/objdump: build ID mismatch
	STALE cmd/pack: build ID mismatch
	STALE cmd/pprof: build ID mismatch
	STALE cmd/test2json: build ID mismatch
	STALE cmd/trace: build ID mismatch
	STALE cmd/vet: build ID mismatch

CC @cagedmantis @dmitshur @toothrot

@gopherbot
Copy link

@gopherbot gopherbot commented Mar 18, 2020

Change https://golang.org/cl/223925 mentions this issue: cmd/go: rebuild cmd/cgo in test_race_install_cgo instead of asserting that it is not stale

@bcmills bcmills changed the title cmd/dist: "unexpected stale targets" during toolchain3 bootstrapping on darwin and solaris builders cmd/dist: "unexpected stale targets" on darwin builders Mar 18, 2020
@bcmills bcmills removed the WaitingForInfo label Mar 18, 2020
@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 18, 2020

I haven't seen the solaris failure mode in months, so retitling to focus on the remaining darwin failures.

@bcmills bcmills added the Builders label Mar 18, 2020
@bcmills bcmills changed the title cmd/dist: "unexpected stale targets" on darwin builders x/build: "unexpected stale targets" on darwin builders Mar 18, 2020
@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 18, 2020

The affected builders seem to be:

  • darwin-amd64-10_12
  • darwin-amd64-10_14
  • darwin-amd64-10_15

Curiously, darwin-386-10_14 and darwin-amd64-nocgo appear to be unaffected. To me, that suggests a problem with the builder configuration rather than a bug in cmd/dist.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 18, 2020

(Those builders broke due to unexpected staleness in CL 223746.)

@toothrot
Copy link
Contributor

@toothrot toothrot commented Mar 18, 2020

This could be related to #35459 /cc @cagedmantis

gopherbot pushed a commit that referenced this issue Mar 18, 2020
Some of the darwin-amd64 builders are providing a stale environment.
Let's un-break them while we investigate.

Updates #37573
Updates #33598

Change-Id: I8b79778fe4d5aa916557c1ba89fa9c776d130b01
Reviewed-on: https://go-review.googlesource.com/c/go/+/223925
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Mar 18, 2020

Change https://golang.org/cl/223755 mentions this issue: cmd/dist: run a checkNotStale on the builders before running tests

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 19, 2020

darwin-386-10_14 and darwin-amd64-nocgo seemed unaffected only because they skipped the affected test. (Those configurations do not support the race detector, so go test -race is not meaningful, and thus test_race_install_cgo has nothing to test.)

darwin-amd64-race does not detect cmd staleness in CL 223755 because for some reason the shouldTestCmd function in cmd/dist exempts the cmd binaries from race-mode checking. That seems unwise, but it's not the topic of this issue.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 19, 2020

(Filed the missing race coverage as #37940.)

@bcmills
Copy link
Member Author

@bcmills bcmills commented Jun 6, 2020

Another clue, from CL 236819: if we do not set GOROOT_FINAL_OLD in cmd/dist, then cmd/link/dwarf_test.go also fails due to cmd/link being stale.

https://storage.googleapis.com/go-build-log/b83e1d59/darwin-amd64-10_14_cf83aeff.log

@randall77
Copy link
Contributor

@randall77 randall77 commented Jun 16, 2020

Just happened on a trybot build, CL 238077, trybot run on patchset 6. linux/arm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.