x/vgo: panic: unbuildable

[rogpeppe]

go version devel +104445e Wed Feb 7 19:22:09 2018 +0000 linux/amd64 vgo:2018-02-20.1

Somehow this go.mod file causes vgo to hit a TODO panic in golang.org/x/vgo/vendor/cmd/go/internal/mvs/mvs.go:

module "github.com/juju/juju"

require "github.com/juju/description" v0.0.0-20180221005849-7ee30d2bc01ef20938adea2efeb991eb817c1f2a

The panic traceback is:

panic: unbuildable

goroutine 1 [running]:
golang.org/x/vgo/vendor/cmd/go/internal/mvs.buildList(0xc420028601, 0x14, 0x0, 0x0, 0x999b60, 0xc42000d2a0, 0x0, 0x0, 0xc42000d2a0, 0xc4201df660, ...)
	/home/rog/src/go/src/golang.org/x/vgo/vendor/cmd/go/internal/mvs/mvs.go:63 +0x103a
golang.org/x/vgo/vendor/cmd/go/internal/mvs.BuildList(0xc420028601, 0x14, 0x0, 0x0, 0x999b60, 0xc42000d2a0, 0x880140, 0xc42029a0b0, 0xe, 0xc420266000, ...)
	/home/rog/src/go/src/golang.org/x/vgo/vendor/cmd/go/internal/mvs/mvs.go:33 +0x78
golang.org/x/vgo/vendor/cmd/go/internal/vgo.iterate(0xc4201df770)
	/home/rog/src/go/src/golang.org/x/vgo/vendor/cmd/go/internal/vgo/load.go:123 +0xae
golang.org/x/vgo/vendor/cmd/go/internal/vgo.importPaths(0xc4200b2010, 0x0, 0x0, 0x77ff77, 0xc42005a2a0, 0x1)
	/home/rog/src/go/src/golang.org/x/vgo/vendor/cmd/go/internal/vgo/load.go:82 +0xe4
golang.org/x/vgo/vendor/cmd/go/internal/vgo.ImportPaths(0xc4200b2010, 0x0, 0x0, 0xc42005a2a0, 0x1, 0x1)
	/home/rog/src/go/src/golang.org/x/vgo/vendor/cmd/go/internal/vgo/load.go:70 +0x5a
golang.org/x/vgo/vendor/cmd/go/internal/load.ImportPaths(0xc4200b2010, 0x0, 0x0, 0x411d89, 0x411d89, 0x16)
	/home/rog/src/go/src/golang.org/x/vgo/vendor/cmd/go/internal/load/pkg.go:1485 +0x4f
golang.org/x/vgo/vendor/cmd/go/internal/load.PackagesAndErrors(0xc4200b2010, 0x0, 0x0, 0xc4201dfa90, 0x4342b9, 0xc420000180)
	/home/rog/src/go/src/golang.org/x/vgo/vendor/cmd/go/internal/load/pkg.go:1457 +0xac
golang.org/x/vgo/vendor/cmd/go/internal/load.PackagesForBuild(0xc4200b2010, 0x0, 0x0, 0x10, 0x10, 0x8b7a20)
	/home/rog/src/go/src/golang.org/x/vgo/vendor/cmd/go/internal/load/pkg.go:1499 +0x5d
golang.org/x/vgo/vendor/cmd/go/internal/work.runBuild(0xbb49c0, 0xc4200b2010, 0x0, 0x0)
	/home/rog/src/go/src/golang.org/x/vgo/vendor/cmd/go/internal/work/build.go:301 +0x85
golang.org/x/vgo/vendor/cmd/go.Main()
	/home/rog/src/go/src/golang.org/x/vgo/vendor/cmd/go/main.go:155 +0x82e
main.main()
	/home/rog/src/go/src/golang.org/x/vgo/main.go:37 +0x25

[mewmew]

I was also hit by an unbuildable panic for the following go.mod of decomp/decomp@b2949c2:

module github.com/decomp/decomp

require (
	bitbucket.org/zombiezen/cardcpx v0.0.0-20150417151802-902f68ff43ef
	github.com/graphism/exp v0.0.0-20180515191528-78b6f6b90bd7
	github.com/graphism/simple v0.0.0-20180515061654-415a775f75dc
	github.com/llir/llvm v0.0.0-20180307193049-784d18aa9fbe
	github.com/mewkiz/pkg v0.0.0-20170428133157-c19a6f9fa6f8
	github.com/pkg/errors v0.8.0
	golang.org/x/exp v0.0.0-20180321215751-8460e604b9de
	gonum.org/v1/gonum v0.0.0-20180523214355-f5d91d72f89a
)

Note, the command I executed was vgo get github.com/decomp/decomp when standing at the root directory of the decomp repository.

[u@x220 decomp]$ vgo get github.com/decomp/decomp
vgo: finding github.com/decomp/decomp v0.0.0-20180524213320-b2949c22ef7f
vgo: finding gonum.org/v1/gonum v0.0.0-20180523214355-f5d91d72f89a
vgo: finding golang.org/x/exp v0.0.0-20180321215751-8460e604b9de
vgo: finding github.com/pkg/errors v0.8.0
vgo: finding github.com/mewkiz/pkg v0.0.0-20170428133157-c19a6f9fa6f8
vgo: finding github.com/llir/llvm v0.0.0-20180307193049-784d18aa9fbe
vgo: finding github.com/graphism/simple v0.0.0-20180515061654-415a775f75dc
vgo: finding github.com/graphism/exp v0.0.0-20180515191528-78b6f6b90bd7
vgo: finding bitbucket.org/zombiezen/cardcpx v0.0.0-20150417151802-902f68ff43ef
panic: unbuildable

goroutine 1 [running]:
golang.org/x/vgo/vendor/cmd/go/internal/mvs.buildList(0xc4201ce520, 0x18, 0x0, 0x0, 0x982c20, 0xc420217040, 0x0, 0x0, 0x412288, 0x50, ...)
	/home/u/goget/src/golang.org/x/vgo/vendor/cmd/go/internal/mvs/mvs.go:63 +0x103a
golang.org/x/vgo/vendor/cmd/go/internal/mvs.BuildList(0xc4201ce520, 0x18, 0x0, 0x0, 0x982c20, 0xc420217040, 0x1, 0x1, 0x0, 0x0, ...)
	/home/u/goget/src/golang.org/x/vgo/vendor/cmd/go/internal/mvs/mvs.go:33 +0x78
golang.org/x/vgo/vendor/cmd/go/internal/mvs.Upgrade(0xc4201ce520, 0x18, 0x0, 0x0, 0x982c60, 0xc4202387e0, 0xc4202387c0, 0x1, 0x1, 0xc420030570, ...)
	/home/u/goget/src/golang.org/x/vgo/vendor/cmd/go/internal/mvs/mvs.go:209 +0x262
golang.org/x/vgo/vendor/cmd/go/internal/vgo.runGet(0xb8fc20, 0xc42009a080, 0x1, 0x1)
	/home/u/goget/src/golang.org/x/vgo/vendor/cmd/go/internal/vgo/get.go:102 +0x829
golang.org/x/vgo/vendor/cmd/go.Main()
	/home/u/goget/src/golang.org/x/vgo/vendor/cmd/go/main.go:155 +0x82e
main.main()
	/home/u/goget/src/golang.org/x/vgo/main.go:31 +0x20

There are no Go files in the root directory, so technically the panic is correct. However, the error reported by go get github.com/decomp/decomp seem more user friendly.

[u@x220 decomp]$ go get github.com/decomp/decomp
can't load package: package github.com/decomp/decomp: no Go files in /tmp/go2/src/github.com/decomp/decomp

vgo build ./... finishes without errors:

[u@x220 decomp]$ vgo build ./...
vgo: downloading github.com/graphism/simple v0.0.0-20180515061654-415a775f75dc
vgo: downloading gonum.org/v1/gonum v0.0.0-20180523214355-f5d91d72f89a
vgo: downloading github.com/llir/llvm v0.0.0-20180307193049-784d18aa9fbe
vgo: downloading github.com/pkg/errors v0.8.0
vgo: downloading golang.org/x/exp v0.0.0-20180321215751-8460e604b9de
vgo: downloading github.com/graphism/exp v0.0.0-20180515191528-78b6f6b90bd7
vgo: downloading bitbucket.org/zombiezen/cardcpx v0.0.0-20150417151802-902f68ff43ef
vgo: downloading github.com/mewkiz/pkg v0.0.0-20170428133157-c19a6f9fa6f8

[anacrolix]

Similar error with github.com/anacrolix/dht. Same stack trace as the others.

[bcmills]

I have a very small repro for @rogpeppe's stack trace using mutually-dependent modules (see #25758).
The panic triggers if the current module is involved in a cycle.

I don't know whether @mewmew's is related.

~$ cd $(mktemp -d)
/tmp/tmp.BVnn6HsRds$ git clone git@github.com:bcmills/vgotest.git
/tmp/tmp.BVnn6HsRds$ cd vgotest
/tmp/tmp.BVnn6HsRds/vgotest$ git checkout a/v0.1.0
/tmp/tmp.BVnn6HsRds/vgotest$ cd a
/tmp/tmp.BVnn6HsRds/vgotest/a$ vgo build .
vgo: finding github.com/bcmills/vgotest/b v0.1.0
vgo: finding github.com/bcmills/vgotest/a v0.1.0
panic: unbuildable

goroutine 1 [running]:
golang.org/x/vgo/vendor/cmd/go/internal/mvs.buildList(0xc0001fa140, 0x1c, 0x0, 0x0, 0x9bb0e0, 0xc000216280, 0x0, 0x0, 0x40dae8, 0x20, ...)
        /usr/local/google/home/bcmills/src/golang.org/x/vgo/vendor/cmd/go/internal/mvs/mvs.go:63 +0x106c
golang.org/x/vgo/vendor/cmd/go/internal/mvs.BuildList(0xc0001fa140, 0x1c, 0x0, 0x0, 0x9bb0e0, 0xc000216280, 0x5a6b4d, 0x87b240, 0x0, 0x0, ...)
        /usr/local/google/home/bcmills/src/golang.org/x/vgo/vendor/cmd/go/internal/mvs/mvs.go:33 +0x78
golang.org/x/vgo/vendor/cmd/go/internal/vgo.iterate(0xc0001e7790)
        /usr/local/google/home/bcmills/src/golang.org/x/vgo/vendor/cmd/go/internal/vgo/load.go:124 +0xb4
golang.org/x/vgo/vendor/cmd/go/internal/vgo.importPaths(0xc00001e0b0, 0x1, 0x1, 0x782c27, 0xc0001fc220, 0x1)
        /usr/local/google/home/bcmills/src/golang.org/x/vgo/vendor/cmd/go/internal/vgo/load.go:83 +0xde
golang.org/x/vgo/vendor/cmd/go/internal/vgo.ImportPaths(0xc00001e0b0, 0x1, 0x1, 0xc0001fc220, 0x1, 0x1)
        /usr/local/google/home/bcmills/src/golang.org/x/vgo/vendor/cmd/go/internal/vgo/load.go:71 +0x58
golang.org/x/vgo/vendor/cmd/go/internal/load.ImportPaths(0xc00001e0b0, 0x1, 0x1, 0x16, 0xc0001fa060, 0x16)
        /usr/local/google/home/bcmills/src/golang.org/x/vgo/vendor/cmd/go/internal/load/pkg.go:1485 +0x4d
golang.org/x/vgo/vendor/cmd/go/internal/load.PackagesAndErrors(0xc00001e0b0, 0x1, 0x1, 0xc0001e7ab0, 0x434f0f, 0xc000000300)
        /usr/local/google/home/bcmills/src/golang.org/x/vgo/vendor/cmd/go/internal/load/pkg.go:1457 +0xa3
golang.org/x/vgo/vendor/cmd/go/internal/load.PackagesForBuild(0xc00001e0b0, 0x1, 0x1, 0x10, 0x8b7560, 0xc000264b20)
        /usr/local/google/home/bcmills/src/golang.org/x/vgo/vendor/cmd/go/internal/load/pkg.go:1499 +0x5d
golang.org/x/vgo/vendor/cmd/go/internal/work.runBuild(0xcc07a0, 0xc00001e0b0, 0x1, 0x1)
        /usr/local/google/home/bcmills/src/golang.org/x/vgo/vendor/cmd/go/internal/work/build.go:301 +0x85
golang.org/x/vgo/vendor/cmd/go.Main()
        /usr/local/google/home/bcmills/src/golang.org/x/vgo/vendor/cmd/go/main.go:155 +0x56e
main.main()
        /usr/local/google/home/bcmills/src/golang.org/x/vgo/main.go:31 +0x20

[leighmcculloch]

I'm seeing this happen on github.com/stretchr/testify.

github.com/stretchr/testify [master] $ vgo build                                                                                 
vgo: finding github.com/stretchr/objx v0.0.0-20180106011353-facf9a85c22f               
vgo: finding github.com/stretchr/testify v1.2.0
vgo: finding github.com/stretchr/objx v0.0.0-20140526180921-cbeaeb16a013                                                                     
vgo: finding github.com/pmezard/go-difflib v0.0.0-20151028094244-d8ed2627bdf0  
vgo: finding github.com/pmezard/go-difflib v1.0.0                                          
vgo: finding github.com/davecgh/go-spew v1.1.0                                                                                               
panic: unbuildable

When I first posted this in #25604 I wasn't clear what was happening, but now I can see that testify imports objx, which in turn in _test.go files imports a specific version (defined by a Gopkg.{toml,lock}) of testify. So there's a circular dependency and it doesn't seem like vgo knows what to do since it wants to build in testify v1.2.0 while building my local copy of testify (master). If I change my local copy to be testify v1.2.0 by checking out that tag, it works.

[gopherbot]

Change https://golang.org/cl/119575 mentions this issue: cmd/go/internal/mvs: test UpgradeAll with transitive requirements

[bcmills]

CL 119575 fixes testify and my toy example.

It removes the panic in the decomp example:

/tmp/tmp.T9opTdmUwY$ git clone https://github.com/decomp/decomp
/tmp/tmp.T9opTdmUwY$ cd decomp
/tmp/tmp.T9opTdmUwY/decomp$ git checkout b2949c2
/tmp/tmp.T9opTdmUwY/decomp$ vgo build .
vgo: import "github.com/decomp/decomp" [/tmp/tmp.T9opTdmUwY/decomp]: no Go source files

I try not to touch anything AGPL at work, so I'll leave it to @rogpeppe to verify whether it fixes juju too. (I suspect that it does.)

[bcmills]

It also fixes the panic for anacrolix/dht, but that package fails to build for other reasons:

./server.go:692:13: triedAddrs.AddString undefined

/tmp/tmp.T9opTdmUwY$ git clone https://github.com/anacrolix/dht.git
Cloning into 'dht'...
remote: Counting objects: 1022, done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 1022 (delta 3), reused 13 (delta 2), pack-reused 1003
Receiving objects: 100% (1022/1022), 327.19 KiB | 3.80 MiB/s, done.
Resolving deltas: 100% (630/630), done.

/tmp/tmp.T9opTdmUwY$ cd dht

/tmp/tmp.T9opTdmUwY/dht$ vgo build .
vgo: creating new go.mod: module github.com/anacrolix/dht
vgo: resolving import "github.com/anacrolix/missinggo/slices"
vgo: finding github.com/anacrolix/missinggo/slices latest
vgo: finding github.com/anacrolix/missinggo latest
vgo: finding github.com/anacrolix/missinggo (latest)
vgo: adding github.com/anacrolix/missinggo v0.0.0-20180614131527-f9a329340c53
vgo: resolving import "github.com/anacrolix/torrent/bencode"
vgo: finding github.com/anacrolix/torrent/bencode latest
vgo: finding github.com/anacrolix/torrent latest
vgo: finding github.com/anacrolix/torrent (latest)
vgo: adding github.com/anacrolix/torrent v0.0.0-20180619130019-62637ce52dbf
vgo: resolving import "github.com/anacrolix/sync"
vgo: finding github.com/anacrolix/sync latest
vgo: finding github.com/anacrolix/sync (latest)
vgo: adding github.com/anacrolix/sync v0.0.0-20180611022320-3c4cb11f5a01
vgo: resolving import "github.com/bradfitz/iter"
vgo: finding github.com/bradfitz/iter latest
vgo: finding github.com/bradfitz/iter (latest)
vgo: adding github.com/bradfitz/iter v0.0.0-20140124041915-454541ec3da2
vgo: resolving import "github.com/willf/bloom"
vgo: finding github.com/willf/bloom v1.0.0
vgo: finding github.com/willf/bloom (latest)
vgo: adding github.com/willf/bloom v1.0.0
vgo: finding github.com/stretchr/objx facf9a85c22f48d2f52f2380e4efce1768749a89
vgo: finding github.com/remyoudompheng/bigfft 52369c62f4463a21c8ff8531194c5526322b8521
vgo: finding github.com/vitrun/qart bf64b92db6b05651d6c25a3dabf2d543b360c0aa
vgo: finding github.com/chmduquesne/rollinghash abb8cbaf9915e48ee20cae94bcd94221b61707a2
vgo: finding github.com/templexxx/xor 0af8e873c554da75f37f2049cdffda804533d44c
vgo: finding github.com/klauspost/reedsolomon 0b30fa71cc8e4e9010c9aba6d0320e2e5b163b29
vgo: finding github.com/d4l3k/messagediff 29f32d820d112dbd66e58492a6ffb7cc3106312b
vgo: finding github.com/calmh/xdr 08e072f9cb164f943a92eb59f90f3abc64ac6e8f
vgo: finding github.com/minio/sha256-simd ad98a36ba0da87206e3378c556abbfeaeaa98668
vgo: finding github.com/onsi/ginkgo 6c46eb8334b30dc55b42f1a1c725d5ce97375390
vgo: finding github.com/jackpal/gateway 5795ac81146e01d3fab7bcf21c043c3d6a32b006
vgo: finding github.com/gogo/protobuf 160de10b2537169b5ae3e7e221d28269ef40d311
vgo: finding github.com/cheggaaa/pb 18d384da9bdc1e5a08fc2a62a494c321d9ae74ea
vgo: finding github.com/BurntSushi/toml a368813c5e648fee92e5f6c30e3944ff9d5e8895
vgo: finding github.com/prometheus/common 2e54d0b93cba2fd133edc32211dcc32c06ef72ca
vgo: finding github.com/a8m/mark 44f2db6188458162890ca13980819247418d8e45
vgo: finding github.com/oschwald/maxminddb-golang 26fe5ace1c706491c2936119e1dc69c1a9c04d7f
vgo: finding github.com/mitchellh/go-homedir b8bc1bf767474819792c23f32d8286a45736f1c6
vgo: finding github.com/go-ini/ini 32e4c1e6bc4e7d0d8451aa6b75200d19e37a536a
vgo: finding github.com/stathat/go 74669b9f388d9d788c97399a0824adbfee78400e
vgo: finding github.com/gobwas/glob 51eb1ee00b6d931c66d229ceeb7c31b985563420
vgo: finding github.com/bkaradzic/go-lz4 7224d8d8f27ef618c0a95f1ae69dbb0488abc33a
vgo: finding golang.org/x/sys 83801418e1b59fb1880e363299581ee543af32ca
vgo: finding github.com/mattn/go-runewidth 97311d9f7767e3d6f422ea06661bc2c7a19e8a5d
vgo: finding golang.org/x/net d866cfc389cec985d6fda2859936a575a55a3ab6
vgo: finding github.com/minio/cli 45db1f8a055198ad8c12754026cb2c51c584c756
vgo: finding github.com/templexxx/cpufeat 3794dfbfb04749f896b521032f69383f24c3687e
vgo: finding github.com/oschwald/geoip2-golang 5b1dc16861f81d05d9836bb21c2d0d65282fc0b8
vgo: finding github.com/calmh/du dd9dc2043353249b2910b29dcfd6f6d4e64f39be
vgo: finding github.com/minio/minio-go 17b9efe2ee358a550ff2d414160b75fc85c86f2e
vgo: finding github.com/prometheus/client_golang 180b8fdc22b4ea7750bcb43c925277654a1ea2f3
vgo: finding github.com/petermattis/goid 3db12ebb2a599ba4a96bea1c17b61c2f78a40e02
vgo: finding github.com/AudriusButkevicius/cli 7f561c78b5a4aad858d9fd550c92b5da6d55efbb
vgo: finding github.com/AudriusButkevicius/pfilter 9dca34a5b530bfc9843fa8aa2ff08ff9821032cb
vgo: finding golang.org/x/crypto 95a4943f35d008beabde8c11e5075a1b714e6419
vgo: finding github.com/golang/snappy 553a641470496b2327abcac10b36396bd98e45c9
vgo: finding github.com/dustin/go-humanize bb3d318650d48840a39aa21a027c6630e198e626
vgo: finding github.com/AudriusButkevicius/go-nat-pmp 452c97607362b2ab5a7839b8d1704f0396b640ca
vgo: finding github.com/golang/groupcache 84a468cf14b4376def5d68c722b139b881c450a4
vgo: finding github.com/kardianos/osext ae77be60afb1dcacde03767a8c37337fad28ac14
vgo: finding github.com/sirupsen/logrus d682213848ed68c0a260ca37d6dd5ace8423f5ba
vgo: finding gopkg.in/urfave/cli.v1 cfb38830724cc34fedffe9a2a29fb54fa9169cd1
vgo: finding github.com/syndtr/goleveldb 34011bf325bce385408353a30b101fe5e923eb6e
vgo: finding github.com/kballard/go-shellquote cd60e84ee657ff3dc51de0b4f55dd299a3e136f2
vgo: finding golang.org/x/text e19ae1496984b1c655b8044a65c0300a3c878dd3
vgo: finding github.com/gernest/wow 7e0b2a2398989a5d220eebac5742d45422ba7de8
vgo: finding code.cloudfoundry.org/bytefmt a052d587819f45f719a22e344a8ad7858deb3733
vgo: finding github.com/thejerf/suture 87e298c9891673c9ae76e10c2c9be589127e5f49
vgo: finding github.com/syncthing/notify b9ceffc925039c77cd9e0d38f248279ccc4399e2
vgo: finding golang.org/x/time 6dc17368e09b0e8634d71cac8168d853e869a0c7
vgo: finding golang.org/x/crypto 0fcca4842a8d74bfddc2c96a073bd2a4d2a7a2e8
vgo: finding github.com/rcrowley/go-metrics e181e095bae94582363434144c61a9653aff6e50
vgo: finding github.com/onsi/gomega ba3724c94e4dd5d5690d37c190f1c54b2c1b4e64
vgo: finding github.com/magefile/mage 63768081a3236a7c6c53ef72e402ae1fe1664b61
vgo: finding github.com/prometheus/procfs b15cd069a83443be3154b719d0cc9fe8117f09fb
vgo: finding github.com/golang/protobuf 1e59b77b52bf8e4b449a57e6f79f21226d571845
vgo: finding github.com/prometheus/client_model 99fa1f4be8e564e8a6b613da7fa6f46c9edafc6c
vgo: finding github.com/edsrzf/mmap-go 0bce6a6887123b67a60366d2c9fe2dfb74289d2e
vgo: finding github.com/sasha-s/go-deadlock 03d40e5dbd5488667a13b3c2600b2f7c2886f02f
vgo: finding github.com/klauspost/cpuid eae9b3e628d72774e13bdf024e78c0802f85a5b9
vgo: finding github.com/pkg/errors e881fd58d78e04cf6d0de1217f8707c8cc2249bc
vgo: finding gopkg.in/yaml.v2 287cf08546ab5e7e37d55a84f7ed3fd1db036de5
vgo: finding github.com/tjfoc/gmsm 98aa888b79d8de04afe0fccf45ed10594efc858b
vgo: downloading github.com/anacrolix/missinggo v0.0.0-20180614131527-f9a329340c53
vgo: downloading github.com/bradfitz/iter v0.0.0-20140124041915-454541ec3da2
vgo: downloading github.com/anacrolix/torrent v0.0.0-20180619130019-62637ce52dbf
vgo: downloading github.com/huandu/xstrings v1.0.0
vgo: downloading github.com/anacrolix/sync v0.0.0-20180611022320-3c4cb11f5a01
vgo: downloading github.com/edsrzf/mmap-go v0.0.0-20170320065105-0bce6a688712
vgo: downloading github.com/willf/bloom v1.0.0
vgo: downloading github.com/willf/bitset v1.1.3
# github.com/anacrolix/dht
./server.go:692:13: triedAddrs.AddString undefined (type *bloom.BloomFilter has no field or method AddString)

[anacrolix]

@bcmills It's because vgo defaults to choosing v1.0.0. If I manually change it to v2.0.0 in this case, it builds. This is what it's generating anacrolix/dht@920ba28 now.

[gopherbot]

Change https://golang.org/cl/120996 mentions this issue: cmd/go/internal/mvs: ignore requirements for target module