GOROOT job failures, temporally linked with gvm 1.0.21 deployment #2553

Closed
ericlemar opened this Issue Jul 23, 2014 · 19 comments

Comments

Projects
None yet
6 participants

Is there some general breakage with go1.3 at the moment? On our tree between two builds we started getting GOROOT errors at the very top of our jobs (no obvious changes of ours to cause that), but the Go Version Manager seems to have bumped to v1.0.21 from v1.0.20 at exactly this point. Relevent portions of the output of post/pre are below.

Broken:

$ gvm version
Go Version Manager v1.0.21 installed at /home/travis/.gvm
$ go version
go version go1.3 linux/amd64
go.env
$ go env
GOARCH="amd64"
GOBIN=""
GOCHAR="6"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/travis/gopath:/home/travis/.gvm/pkgsets/go1.3/global"
GORACE=""
GOROOT="/home/travis/.gvm/gos/go1.3"
GOTOOLDIR="/home/travis/.gvm/gos/go1.3/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

....

$ true
$ make test dist
./env.sh go install -v igneous.io/common/...
go: cannot find GOROOT directory: /usr/local/go
make: *** [build_common] Error 2


Working:

$ gvm version
Go Version Manager v1.0.20 installed at /home/travis/.gvm
$ go version
go version go1.3 linux/amd64
go.env
$ go env
GOARCH="amd64"
GOBIN=""
GOCHAR="6"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/travis/gopath:/home/travis/.gvm/pkgsets/go1.3/global"
GORACE=""
GOROOT="/home/travis/.gvm/gos/go1.3"
GOTOOLDIR="/home/travis/.gvm/gos/go1.3/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"
.....

$ make test dist
./env.sh go install -v igneous.io/common/...
github.com/crowdmob/goamz/aws

Owner

joshk commented Jul 23, 2014

@meatballhat I'm pretty sure some recent changes just borked GVM :/

Owner

joshk commented Jul 23, 2014

@ericlemar thank you for reporting this!

Owner

meatballhat commented Jul 23, 2014

Definitely seems fishy. I'm trying to get gvm to run all of its tests in Travis to hopefully suss out the problem(s). Thanks for patience!

/cc @jbussdeiker

codemac commented Jul 24, 2014

Any progress or new information on this issue?

Owner

meatballhat commented Jul 24, 2014

Hi Jeff, I didn't get to look at this last night like I'd planned.
Hopefully tonight. Do you feel like there's enough information here for me
to reproduce the bug?
On Jul 24, 2014 5:36 PM, "Jeff Mickey" notifications@github.com wrote:

Any progress or new information on this issue?


Reply to this email directly or view it on GitHub
#2553 (comment)
.

codemac commented Jul 24, 2014

From what I can tell - you should be able to repro it as this had 0 code change on our part. The key here is that GOROOT from go env earlier is going to a gvm directory (/home/travis/.gvm/gos/go1.3) and then once we run go install, GOROOT is back to something much more common (/usr/local/go). Seems like the environment in the travis run isn't sticking around, based on my very limited understanding of gvm.

If you have any questions or can't repro it, please comment here and eric and I will definitely help out.

Owner

meatballhat commented Jul 24, 2014

@codemac Got it, thank you!

Owner

meatballhat commented Jul 24, 2014

@codemac I'm probably missing something obvious, but to what project/repo does the above output belong?

codemac commented Jul 25, 2014

It's a private organization / repo on github

Owner

meatballhat commented Jul 27, 2014

@codemac Just looking at this now. Sorry about the delay in communication. Since it's a private repo, are you able to provide me with the full .travis.yml with secret bits redacted? I'm going to poke at this for a bit anyway, but just in case that's fruitless...

A redacted version of the .yml file

language: go
go:
- 1.3
addons:
  postgresql: '9.3'
notifications:
  email:
    recipients: ABCD@XXXX
  webhooks:
    urls:
      - https://XXX/YYY
    on_start: true
before_install:
- npm install -g grunt-cli
script:
- make test dist
after_success:
- make dist
deploy:
  provider: s3
  access_key_id: XXXXXXXXXXXXXXX
  secret_access_key:
    secure: XXXX/YYYY
  bucket: XXXXXX
  upload-dir: NNN
  local-dir: dist/artifacts
  skip_cleanup: true
  on:
    all_branches: true
Owner

joshk commented Jul 28, 2014

Hey @ericlemar

Is it possible to try to replicate this issue on travis-ci.org?

That way there is a project for @meatballhat to fork and play with?

Owner

meatballhat commented Jul 28, 2014

@ericlemar As Josh said, anything you can do to help me reproduce this is greatly appreciated 😺

I've had no luck so far on this toy repo: https://github.com/meatballhat/yolo-octo-adventure

henrikhodne added the go label Jul 29, 2014

codemac commented Aug 25, 2014

This disappeared for a short while, but it's back again today with the gvm 1.0.22 update.. same repo and everything as above.

It once again ignores it's own GOROOT setting..

From our travis build:

 * Starting PostgreSQL 9.3 database server

gvm.get

1.11s$ gvm get
Your branch is up-to-date with 'origin/master'.
remote: Counting objects: 65, done.
remote: Compressing objects: 100% (32/32), done.
remote: Total 65 (delta 31), reused 53 (delta 27)
Unpacking objects: 100% (65/65), done.
From https://github.com/moovweb/gvm
b914f5a..e5a06b5 master -> origin/master
* [new tag] 1.0.22 -> 1.0.22
From https://github.com/moovweb/gvm
* [new tag] 1.0.21 -> 1.0.21
Updating b914f5a..e5a06b5
Fast-forward
.travis.yml | 9 +++--
README | 2 ++
README.md | 4 ++-
Rakefile | 42 ++++++++++++++--------
VERSION | 2 +-
binscripts/gvm-installer | 22 ++++++------
scripts/cross | 6 ++--
scripts/gvm-check | 36 ++++++++++++-------
scripts/install | 35 +++++++++---------
.../gvm_install_1.2_preferbinary_comment_test.sh | 2 +-
10 files changed, 99 insertions(+), 61 deletions(-)

gvm.update

13.25s$ gvm update && source $HOME/.gvm/scripts/gvm
pulling from https://go.googlecode.com/hg/
searching for changes
adding changesets
adding manifests
adding file changes
added 501 changesets with 1553 changes to 806 files (+1 heads)
(run 'hg heads' to see heads)

gvm.install

4.47s$ gvm install go1.3.1 --binary || gvm install go1.3.1
Installing go1.3.1 from binary source
0.05s$ gvm use go1.3.1
Now using version go1.3.1
$ export GOPATH=$HOME/gopath:$GOPATH
$ mkdir -p $HOME/gopath/src/github.com/igneous-systems
$ cp -r $TRAVIS_BUILD_DIR $HOME/gopath/src/github.com/igneous-systems/mesa
$ export TRAVIS_BUILD_DIR=$HOME/gopath/src/github.com/igneous-systems/mesa
$ cd $HOME/gopath/src/github.com/igneous-systems/mesa
0.07s$ gvm version
Go Version Manager v1.0.22 installed at /home/travis/.gvm
0.02s$ go version
go version go1.3.1 linux/amd64

go.env

0.02s$ go env
GOARCH="amd64"
GOBIN=""
GOCHAR="6"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/travis/gopath:/home/travis/.gvm/pkgsets/go1.3.1/global"
GORACE=""
GOROOT="/home/travis/.gvm/gos/go1.3.1"
GOTOOLDIR="/home/travis/.gvm/gos/go1.3.1/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

before_install

2.47s$ npm install -g grunt-cli
npm http GET https://registry.npmjs.org/grunt-cli
npm http 200 https://registry.npmjs.org/grunt-cli
npm http GET https://registry.npmjs.org/grunt-cli/-/grunt-cli-0.1.13.tgz
npm http 200 https://registry.npmjs.org/grunt-cli/-/grunt-cli-0.1.13.tgz
npm http GET https://registry.npmjs.org/resolve
npm http GET https://registry.npmjs.org/nopt
npm http GET https://registry.npmjs.org/findup-sync
npm http 200 https://registry.npmjs.org/findup-sync
npm http GET https://registry.npmjs.org/findup-sync/-/findup-sync-0.1.3.tgz
npm http 200 https://registry.npmjs.org/nopt
npm http 200 https://registry.npmjs.org/resolve
npm http 200 https://registry.npmjs.org/findup-sync/-/findup-sync-0.1.3.tgz
npm http GET https://registry.npmjs.org/nopt/-/nopt-1.0.10.tgz
npm http GET https://registry.npmjs.org/resolve/-/resolve-0.3.1.tgz
npm http 200 https://registry.npmjs.org/nopt/-/nopt-1.0.10.tgz
npm http 200 https://registry.npmjs.org/resolve/-/resolve-0.3.1.tgz
npm http GET https://registry.npmjs.org/abbrev
npm http GET https://registry.npmjs.org/glob
npm http GET https://registry.npmjs.org/lodash
npm http 200 https://registry.npmjs.org/abbrev
npm http 200 https://registry.npmjs.org/lodash
npm http GET https://registry.npmjs.org/abbrev/-/abbrev-1.0.5.tgz
npm http GET https://registry.npmjs.org/lodash/-/lodash-2.4.1.tgz
npm http 200 https://registry.npmjs.org/glob
npm http GET https://registry.npmjs.org/glob/-/glob-3.2.11.tgz
npm http 200 https://registry.npmjs.org/abbrev/-/abbrev-1.0.5.tgz
npm http 200 https://registry.npmjs.org/glob/-/glob-3.2.11.tgz
npm http 200 https://registry.npmjs.org/lodash/-/lodash-2.4.1.tgz
npm http GET https://registry.npmjs.org/inherits
npm http GET https://registry.npmjs.org/minimatch
npm http 200 https://registry.npmjs.org/minimatch
npm http GET https://registry.npmjs.org/minimatch/-/minimatch-0.3.0.tgz
npm http 200 https://registry.npmjs.org/inherits
npm http GET https://registry.npmjs.org/inherits/-/inherits-2.0.1.tgz
npm http 200 https://registry.npmjs.org/minimatch/-/minimatch-0.3.0.tgz
npm http 200 https://registry.npmjs.org/inherits/-/inherits-2.0.1.tgz
npm http GET https://registry.npmjs.org/lru-cache
npm http GET https://registry.npmjs.org/sigmund
npm http 200 https://registry.npmjs.org/sigmund
npm http GET https://registry.npmjs.org/sigmund/-/sigmund-1.0.0.tgz
npm http 200 https://registry.npmjs.org/sigmund/-/sigmund-1.0.0.tgz
npm http 200 https://registry.npmjs.org/lru-cache
npm http GET https://registry.npmjs.org/lru-cache/-/lru-cache-2.5.0.tgz
npm http 200 https://registry.npmjs.org/lru-cache/-/lru-cache-2.5.0.tgz
/home/travis/.nvm/v0.10.28/bin/grunt -> /home/travis/.nvm/v0.10.28/lib/node_modules/grunt-cli/bin/grunt
grunt-cli@0.1.13 /home/travis/.nvm/v0.10.28/lib/node_modules/grunt-cli
├── resolve@0.3.1
├── nopt@1.0.10 (abbrev@1.0.5)
└── findup-sync@0.1.3 (lodash@2.4.1, glob@3.2.11)

install

0.01s$ true
0.07s$ make test dist
./env.sh go install -v igneous.io/common/...
go: cannot find GOROOT directory: /usr/local/go
make: *** [build_common] Error 2
The command "make test dist" exited with 2.
Done. Your build exited with 1.
Owner

meatballhat commented Aug 28, 2014

@codemac Any chance you can share the content of your Makefile and ./env.sh?

codemac commented Sep 2, 2014

OK, figured a few things out:

  • our env.sh script was unsetting all environment variables that start with GO* which included GOROOT
  • That is bad.

Sorry for the distraction. I don't think this goroot unsetting was there initially, but I'm not sure.

Owner

meatballhat commented Oct 2, 2014

@codemac Glad you found it! 👏 Sorry for the crazy long feedback loop 😺 Can this issue be closed, in your opinion?

codemac commented Oct 2, 2014

Yes, close this out. I'm curious what ways would have prevented this, I'm not sure I have a great answer.

BanzaiMan closed this Oct 2, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment