Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: aix-ppc64 builder has double/unmerged PATH causing wrong 'go' binary to be found #31567

Open
Helflym opened this issue Apr 19, 2019 · 13 comments
Open

x/build: aix-ppc64 builder has double/unmerged PATH causing wrong 'go' binary to be found #31567

Helflym opened this issue Apr 19, 2019 · 13 comments
Labels
Builders help wanted NeedsFix OS-AIX
Milestone
Unreleased

Comments

@Helflym
Copy link
Contributor

@Helflym Helflym commented Apr 19, 2019 •
edited by dmitshur

The tests of x/tools and x/tour are failing on aix/ppc64, because the builder seems to use an hybrid version between 1.12.1 and master. Any idea of what must be changed/fixed ?

https://build.golang.org/log/b7b160e7a7e733fbe768d45af70cf1c626ac4289
https://build.golang.org/log/7e28abbbd860c0ea1f212805b15ce5f3e8bf5d14

/cc @bradfitz

Edit by @dmitshur: The root cause of this problem is described in #31567 (comment).
@gopherbot gopherbot added this to the Unreleased milestone Apr 19, 2019
@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Apr 19, 2019

I don't know what's going on there.

Per the first line of that first URL, it should only be using Go built at c8aaec2, which you see later in the log with:

# internal/cpu
compile: version "devel c8aaec2f70c5ccbca1ec2152c57d19981ac09133" does not match go tool version "go1.12.1"

So I don't know why it's expecting Go 1.12.1 or where that comes from.

/cc @ianlancetaylor @bcmills @dmitshur

@bcmills
Copy link
Member

@bcmills bcmills commented Apr 19, 2019 •
edited

There are two PATH entries in the command line logged at the top:

"PATH=/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/bin:/opt/freeware/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin"

and

"PATH=/opt/freeware/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin"

The latter does not include $GOROOT/bin, so that's presumably where the go at 1.12.1 is coming from.

I don't know why compile is coming from the devel version, though.

@Helflym
Copy link
Contributor Author

@Helflym Helflym commented Apr 19, 2019

I think the second path comes from https://go-review.googlesource.com/c/build/+/152457. However, they are supposed to be merged together or at least $GOROOT/bin should be added to the $PATH coming from the dashboard.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Apr 19, 2019

I don't know why compile is coming from the devel version, though.

Because this is a build that's running repo "go" at master (rev c8aaec2f70c5ccbca1ec2152c57d19981ac09133) against x/net master.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Apr 19, 2019

But, yeah, double/unmerged PATH definitely looks like the problem here. Thanks for noticing that and for the CL link.

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 29, 2019

Change https://golang.org/cl/192327 mentions this issue: dashboard: disable failing repos on misconfigured builders

gopherbot pushed a commit to golang/build that referenced this issue Aug 29, 2019
@bcmills
dashboard: disable failing repos on misconfigured builders
5a2c611 
Updates golang/go#32836
Updates golang/go#31567
Updates golang/go#11811

Change-Id: I5443b61cf7732abf906ce2e93eca5408579a55c8
Reviewed-on: https://go-review.googlesource.com/c/build/+/192327
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@bcmills bcmills changed the title x/tools, x/tour: wrong builder version on aix/ppc64 x/build: 'go' subprocesses invoke mismatched versions on aix-ppc64 builder Aug 29, 2019
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 29, 2019

Change https://golang.org/cl/192335 mentions this issue: dashboard: extend aix-ppc64 skip to the website repo

gopherbot pushed a commit to golang/build that referenced this issue Aug 29, 2019
@bcmills
dashboard: extend aix-ppc64 skip to the website repo
485f930 
Updates golang/go#31567

Change-Id: Ia8d996a70166c4395393f4674af87ad755104fe1
Reviewed-on: https://go-review.googlesource.com/c/build/+/192335
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 30, 2019

Change https://golang.org/cl/192559 mentions this issue: go.mod: add go version directive

gopherbot pushed a commit to golang/website that referenced this issue Aug 30, 2019
@bcmills
go.mod: add go version directive
6a9ae81 
(I mostly just wanted to make a trivial change to this repo to clear
out the failure cells for misconfigured builders on the landing page
of https://build.golang.org.)

Updates golang/go#31567

Change-Id: I2d779a143c711e86fb2f34451d7398d3514623ae
Reviewed-on: https://go-review.googlesource.com/c/website/+/192559
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@dmitshur dmitshur changed the title x/build: 'go' subprocesses invoke mismatched versions on aix-ppc64 builder x/build: aix-ppc64 builder has double/unmerged PATH causing wrong 'go' binary to be found Sep 21, 2019
codebien added a commit to codebien/build that referenced this issue Nov 13, 2019
@bcmills @codebien
dashboard: disable failing repos on misconfigured builders
382f633 
Updates golang/go#32836
Updates golang/go#31567
Updates golang/go#11811

Change-Id: I5443b61cf7732abf906ce2e93eca5408579a55c8
Reviewed-on: https://go-review.googlesource.com/c/build/+/192327
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
codebien added a commit to codebien/build that referenced this issue Nov 13, 2019
@bcmills @codebien
dashboard: extend aix-ppc64 skip to the website repo
355857d 
Updates golang/go#31567

Change-Id: Ia8d996a70166c4395393f4674af87ad755104fe1
Reviewed-on: https://go-review.googlesource.com/c/build/+/192335
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bcmills
Copy link
Member

@bcmills bcmills commented Feb 20, 2020 •
edited

https://build.golang.org/log/18084f24be9cc1f31dbef14b8e398003e21045e0

--- FAIL: TestHookChangeGofmt (2.94s)
    hook_test.go:267: invoking commit hook explicitly
    hook_test.go:268: git-codereview hook-invoke pre-commit
    hook_test.go:271: change without hook installed
    hook_test.go:273: git-codereview change
    hook_test.go:276: change with hook installed
    hook_test.go:493: in git-codereview/, ran [go build -o /ramdisk8GB/workdir-host-aix-ppc64-osuosl/tmp/git-codereview-test399063055/git-client/git-codereview]: exit status 2
        # unicode
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # crypto/internal/subtle
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # runtime/internal/sys
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # math/bits
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # unicode/utf8
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # vendor/golang.org/x/crypto/internal/subtle
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # internal/cpu
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # encoding
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # crypto/subtle
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # internal/nettrace
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # container/list
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # unicode/utf16
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # internal/race
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # vendor/golang.org/x/crypto/cryptobyte/asn1
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # runtime/internal/atomic
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # sync/atomic
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # runtime/cgo
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
--- FAIL: TestHookCommitMsgFromGit (2.54s)
    hook_test.go:493: in git-codereview/, ran [go build -o /ramdisk8GB/workdir-host-aix-ppc64-osuosl/tmp/git-codereview-test792055703/git-client/git-codereview]: exit status 2
        # vendor/golang.org/x/crypto/internal/subtle
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # internal/cpu
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # unicode/utf16
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # crypto/subtle
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # container/list
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # vendor/golang.org/x/crypto/cryptobyte/asn1
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # encoding
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # internal/nettrace
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # unicode/utf8
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # runtime/internal/sys
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # crypto/internal/subtle
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # internal/race
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # math/bits
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # unicode
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # runtime/internal/atomic
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # sync/atomic
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
        # runtime/cgo
        compile: version "devel 49f8d45994b6782e4925057909e4896832334a26" does not match go tool version "go1.13"
FAIL
FAIL	golang.org/x/review/git-codereview	72.131s

CC @cagedmantis @toothrot

@bcmills bcmills mentioned this issue Aug 31, 2020
Closed
@bcmills
Copy link
Member

@bcmills bcmills commented Oct 8, 2021

As of CL 353549 the PATH is no longer doubled and the failure mode of this builder no longer mixes go components from different parts of the path.

It's still failing, but it's now at least failing in a much more coherent way that indicates a completely-stale go toolchain rather than just a partly-stale one (https://build.golang.org/log/f18e14391e30e30a150c41a4dd79ec9e6a22f02b):

:: Running /ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/bin/go with args ["/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/bin/go" "test" "-short" "golang.org/x/review/..."] and env ["TERM=dumb" "AUTHSTATE=compat" "SHELL=/usr/bin/ksh" "HOME=/" "USER=root" "PATH=/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/bin:/opt/freeware/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin" "TZ=CST6CDT" "LANG=en_US" "LOCPATH=/usr/lib/nls/loc" "LC__FASTMSG=true" "ODMDIR=/etc/objrepos" "CLCMD_PASSTHRU=1" "LOGNAME=root" "LOGIN=root" "META_BUILDLET_BINARY_URL=https://storage.googleapis.com/go-builder-data/buildlet.aix-ppc64" "GO_STAGE0_NET_DELAY=400ms" "GO_STAGE0_DL_DELAY=100ms" "NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat:/usr/lib/nls/msg/%l.%c/%N:/usr/lib/nls/msg/%l.%c/%N.cat" "WORKDIR=/ramdisk8GB/workdir-host-aix-ppc64-osuosl" "GOROOT_BOOTSTRAP=/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go1.4" "GO_BUILDER_NAME=aix-ppc64" "GOROOT_BOOTSTRAP=/opt/freeware/lib/golang" "PATH=/opt/freeware/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin" "GOROOT=/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go" "GOPATH=/ramdisk8GB/workdir-host-aix-ppc64-osuosl/gopath" "GOPROXY=http://gk3-services-nap-yh2ma9qv-7a84e634-8cll.c.symbolic-datum-552.internal:30157" "GOPROXY=https://proxy.golang.org" "TMPDIR=/ramdisk8GB/workdir-host-aix-ppc64-osuosl/tmp" "GOCACHE=/ramdisk8GB/workdir-host-aix-ppc64-osuosl/gocache"] in dir /ramdisk8GB/workdir-host-aix-ppc64-osuosl/gopath/src/golang.org/x/review
…
--- FAIL: TestHookCommitMsgFromGit (0.69s)
    hook_test.go:493: in git-codereview/, ran [go build -o /ramdisk8GB/workdir-host-aix-ppc64-osuosl/tmp/git-codereview-test3736945410/git-client/git-codereview]: exit status 1
        api.go:15:2: //go:build comment without // +build comment
FAIL
FAIL	golang.org/x/review/git-codereview	72.903s

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 8, 2021

Change https://golang.org/cl/354754 mentions this issue: cmd/buildlet: in setPathEnv, override the last PATH entry, not the first

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 8, 2021

Change https://golang.org/cl/354755 mentions this issue: dashboard: re-enable default repos on aix

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Oct 20, 2021

@bcmills Based on a recent test failure (e.g., https://build.golang.org/log/3d50d2e41477382548e0b84905d5dff6ea05f2f2) and the test code, to me it looks like the problem may be that the test code tries to run "go" from PATH, but the Go toolchain that was built and is being tested (the one in GOROOT/bin) is not in PATH.

I'm only seeing one PATH entry in https://build.golang.org/log/3d50d2e41477382548e0b84905d5dff6ea05f2f2, so though CL 354754 looks like a good change to me, it may not be enough.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders help wanted NeedsFix OS-AIX
Projects
None yet
Milestone
Unreleased
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
@bradfitz @dmitshur @bcmills @gopherbot @Helflym