Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: freebsd-arm-paulzhol builder is failing cmd/link TestDWARF #41101

Open
paulzhol opened this issue Aug 28, 2020 · 3 comments
Open

x/build: freebsd-arm-paulzhol builder is failing cmd/link TestDWARF #41101

paulzhol opened this issue Aug 28, 2020 · 3 comments

Comments

@paulzhol
Copy link
Member

@paulzhol paulzhol commented Aug 28, 2020

https://build.golang.org/log/d0025d71cbb3628b7800d2eebe1d8fb58c79969d

ok  	cmd/internal/src	0.034s
ok  	cmd/internal/test2json	1.729s
--- FAIL: TestDWARF (4.86s)
    dwarf_test.go:41: cmd/link is stale - run go install cmd/link
FAIL
FAIL	cmd/link	56.431s
FAIL
go tool dist: Failed: exit status 1

I've captured the stdout/stderr output from the buildlet to look at the env. variables as I understand it was related to clearing GOROOT_FINAL in similar issues.

2020/08/28 11:21:06 [0x314ba580] Running /tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go with args ["/tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go" "tool" "dist" "test" "--no-rebuild" "--banner=XXXBANNERXXX:" "go_test:cmd/internal/obj/riscv" "go_test:cmd/internal/obj/x86" "go_test:cmd/internal/objabi"] and env ["BLOCKSIZE=K" "MAIL=/var/mail/paulzhol" "PATH=/tmp/workdir-host-freebsd-arm-paulzhol/go/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/paulzhol/bin" "GO_TEST_TIMEOUT_SCALE=30" "CGO_ENABLED=1" "GOARM=7" "XDG_CACHE_HOME=/tmp/cache" "USER=paulzhol" "HOME=/home/paulzhol" "PWD=/" "RC_PID=21" "WORKDIR=/tmp/workdir-host-freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/tmp/workdir-host-freebsd-arm-paulzhol/go1.4" "GO_BUILDER_NAME=freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4" "GOARM=7" "CGO_ENABLED=1" "GOROOT=/tmp/workdir-host-freebsd-arm-paulzhol/go" "GOPATH=/tmp/workdir-host-freebsd-arm-paulzhol/gopath" "GOPROXY=http://10.240.0.3:30157" "GOPROXY=off" "TMPDIR=/tmp/workdir-host-freebsd-arm-paulzhol/tmp" "GOCACHE=/tmp/workdir-host-freebsd-arm-paulzhol/gocache"] in dir /tmp/workdir-host-freebsd-arm-paulzhol
2020/08/28 11:22:09 [0x314ba580] Run = ok, after 1m3.510284157s

2020/08/28 11:22:09 [0x314bb1e0] Running /tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go with args ["/tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go" "tool" "dist" "test" "--no-rebuild" "--banner=XXXBANNERXXX:" "go_test:cmd/internal/src" "go_test:cmd/internal/test2json" "go_test:cmd/link"] and env ["BLOCKSIZE=K" "MAIL=/var/mail/paulzhol" "PATH=/tmp/workdir-host-freebsd-arm-paulzhol/go/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/paulzhol/bin" "GO_TEST_TIMEOUT_SCALE=30" "CGO_ENABLED=1" "GOARM=7" "XDG_CACHE_HOME=/tmp/cache" "USER=paulzhol" "HOME=/home/paulzhol" "PWD=/" "RC_PID=21" "WORKDIR=/tmp/workdir-host-freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/tmp/workdir-host-freebsd-arm-paulzhol/go1.4" "GO_BUILDER_NAME=freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4" "GOARM=7" "CGO_ENABLED=1" "GOROOT=/tmp/workdir-host-freebsd-arm-paulzhol/go" "GOPATH=/tmp/workdir-host-freebsd-arm-paulzhol/gopath" "GOPROXY=http://10.240.0.3:30157" "GOPROXY=off" "TMPDIR=/tmp/workdir-host-freebsd-arm-paulzhol/tmp" "GOCACHE=/tmp/workdir-host-freebsd-arm-paulzhol/gocache"] in dir /tmp/workdir-host-freebsd-arm-paulzhol
2020/08/28 11:23:49 [0x314bb1e0] Run = exit status 1, after 1m39.66510577s

2020/08/28 11:23:49 [0x3148cdc0] Running /tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go with args ["/tmp/workdir-host-freebsd-arm-paulzhol/go/bin/go" "tool" "dist" "test" "--no-rebuild" "--banner=XXXBANNERXXX:" "go_test:cmd/link/internal/benchmark" "go_test:cmd/link/internal/ld" "go_test:cmd/link/internal/loader"] and env ["BLOCKSIZE=K" "MAIL=/var/mail/paulzhol" "PATH=/tmp/workdir-host-freebsd-arm-paulzhol/go/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/paulzhol/bin" "GO_TEST_TIMEOUT_SCALE=30" "CGO_ENABLED=1" "GOARM=7" "XDG_CACHE_HOME=/tmp/cache" "USER=paulzhol" "HOME=/home/paulzhol" "PWD=/" "RC_PID=21" "WORKDIR=/tmp/workdir-host-freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/tmp/workdir-host-freebsd-arm-paulzhol/go1.4" "GO_BUILDER_NAME=freebsd-arm-paulzhol" "GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4" "GOARM=7" "CGO_ENABLED=1" "GOROOT=/tmp/workdir-host-freebsd-arm-paulzhol/go" "GOPATH=/tmp/workdir-host-freebsd-arm-paulzhol/gopath" "GOPROXY=http://10.240.0.3:30157" "GOPROXY=off" "TMPDIR=/tmp/workdir-host-freebsd-arm-paulzhol/tmp" "GOCACHE=/tmp/workdir-host-freebsd-arm-paulzhol/gocache"] in dir /tmp/workdir-host-freebsd-arm-paulzhol
2020/08/28 11:23:50 Halting in 1 second.

Maybe this is a dup of #33598 but on FreeBSD-arm.

@cagedmantis
Copy link
Contributor

@cagedmantis cagedmantis commented Aug 31, 2020

@bcmills
Copy link
Member

@bcmills bcmills commented Aug 31, 2020

I refactored that test in CL 236819 for #39385.

I notice that there are two occurrences of GOROOT_BOOTSTRAP in the environment in the linked log — could that be causing the discrepancy? (See also #31567.)

@paulzhol
Copy link
Member Author

@paulzhol paulzhol commented Aug 31, 2020

@bcmills The first one is wrong (GOROOT_BOOTSTRAP=/tmp/workdir-host-freebsd-arm-paulzhol/go1.4) and it is the default set by the buildlets main():

https://github.com/golang/build/blob/1f82dd332eb91abb9335fc2abc4d8039f7ac7ba5/cmd/buildlet/buildlet.go#L255-L264

var inheritedGorootBootstrap string

func initGorootBootstrap() {
	// Remember any GOROOT_BOOTSTRAP to use as a backup in handleExec
	// if $WORKDIR/go1.4 ends up not existing.
	inheritedGorootBootstrap = os.Getenv("GOROOT_BOOTSTRAP")

	// Default if not otherwise configured in dashboard/builders.go:
	os.Setenv("GOROOT_BOOTSTRAP", filepath.Join(*workDir, "go1.4"))
}

the second GOROOT_BOOTSTRAP is coming from the (coordinator?):

https://github.com/golang/build/blob/1e76199ae864f03a22993c156ef26ecc4679e7f4/dashboard/builders.go#L844-L857

// Env returns the environment variables this builder should run with.
func (c *BuildConfig) Env() []string {
	env := []string{"GO_BUILDER_NAME=" + c.Name}
	if c.FlakyNet {
		env = append(env, "GO_BUILDER_FLAKY_NET=1")
	}
	if c.IsLongTest() {
		// Set a private hook in cmd/dist to run main Go repository tests
		// without the default -short flag. See golang.org/issue/12508.
		env = append(env, "GO_TEST_SHORT=0")
	}
	env = append(env, c.HostConfig().env...)
	return append(env, c.env...)
}

from:
https://github.com/golang/build/blob/18a463e644997ff721e2f319f030b5fd63d4739d/dashboard/builders.go#L347

	"host-freebsd-arm-paulzhol": &HostConfig{
		IsReverse:   true,
		ExpectNum:   1,
		Notes:       "Cubiboard2 1Gb RAM dual-core Cortex-A7 (Allwinner A20), FreeBSD 11.1-RELEASE",
		env:         []string{"GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4"},
		OwnerGithub: "paulzhol",
	},

but eventually this is just passed to an os/exec.Cmd struct in the buildlet exec handler, where

// If Env contains duplicate environment keys, only the last
// value in the slice for each duplicate key is used.
// As a special case on Windows, SYSTEMROOT is always added if
// missing and not explicitly set to the empty string.
Env []string

I randomly sampled an active build, and only the correct (last) value is set:

1527  -  S     0:56.71 /var/tmp/workdir-host-freebsd-arm-paulzhol/go/pkg/tool/freebsd_arm/go_bootstrap install -gcflags=all= -ldflags=all= std cmd

[paulzhol@arm-virt-12 ~]$ procstat -e 1527
  PID COMM             ENVIRONMENT
 1527 go_bootstrap     CGO_ENABLED=1 RC_PID=21 GO_BUILDER_NAME=freebsd-arm-paulzhol GO_TEST_TIMEOUT_SCALE=30 USER=paulzhol GOENV=off PWD=/var/tmp/workdir-host-freebsd-arm-paulzhol/go/src HOME=/home/paulzhol GOROOT=/var/tmp/workdir-host-freebsd-arm-paulzhol/go TMPDIR=/tmp/workdir-host-freebsd-arm-paulzhol/tmp MAIL=/var/mail/paulzhol SHLVL=1 WORKDIR=/tmp/workdir-host-freebsd-arm-paulzhol BLOCKSIZE=K XDG_CACHE_HOME=/tmp/cache PATH=/var/tmp/workdir-host-freebsd-arm-paulzhol/go/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/paulzhol/bin GOCACHE=/var/tmp/workdir-host-freebsd-arm-paulzhol/go/pkg/obj/go-build GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4 GOARM=7 _=./cmd/dist/dist TERM=dumb GO386=387 GOARCH=arm GOHOSTARCH=arm GOHOSTOS=freebsd GOOS=freebsd GOMIPS=hardfloat GOMIPS64=hardfloat GOPPC64=power8 GOROOT_FINAL=/var/tmp/workdir-host-freebsd-arm-paulzhol/go LANG=C LANGUAGE=en_US.UTF8 GOPATH=/var/tmp/workdir-host-freebsd-arm-paulzhol/go/pkg/obj/gopath GOBIN= CC=clang
[paulzhol@arm-virt-12 ~]$ ls -l /var/tmp/workdir-host-freebsd-arm-paulzhol/go

Looking at the PATH above, and #31567 (comment) I don't have the bootstrap go1.4 set, or is there any other go installation present.

The other difference I can think of, is my builder has the entire filesystem mounted read-only, so https://golang.org/cl/223746 is needed to ignore failing writes. Although GOROOT=/var/tmp/workdir-host-freebsd-arm-paulzhol/go set above is writable anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.