x/tools/go/packages: vendor is not searched in GOPATH mode

[vektah]

What version of Go are you using (go version)?

$ go version
go version go1.11.5 linux/amd64

Does this issue reproduce with the latest release?

Reproducable on 1.12-rc1

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/vektah/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/vektah/go"
GOPROXY=""
GORACE=""
GOROOT="/home/vektah/local/go"
GOTMPDIR=""
GOTOOLDIR="/home/vektah/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build863238177=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Trying to load a package using x/tools/go/packages doesn't seem to search vendor directories.

For your convenience this code can be fetched from https://github.com/vektah/packages-vendor-test

packages_test.go

package main_test

import (
	"github.com/external/tool"
	"golang.org/x/tools/go/packages"
	"testing"
)

func TestPackageLoading(t *testing.T) {
	// we can reference the vendored package just fine
	if tool.Foo() != true {
		t.Error("vendor imports are broken?")
	}

	// but if we try to load the package via packages
	pkgs, err := packages.Load(nil, "github.com/external/tool")
	if err != nil {
		t.Error(err.Error())
	}

	// we get an error!
	if pkgs[0].Errors != nil {
		t.Error(pkgs[0].Errors)
	}

	// and a blank package name
	if pkgs[0].Name != "tool" {
		t.Error("the package name should be set")
	}
}

vendor/github.com/external/tool/tool.go

package tool

func Foo() bool {
	return true
}

What did you expect to see?

PASS
ok      github.com/vektah/packages-vendor-test  0.001s

What did you see instead?

--- FAIL: TestPackageLoading (0.01s)
    packages_test.go:24: [-: cannot find package "github.com/external/tool" in any of:
                /home/vektah/local/go/src/github.com/external/tool (from $GOROOT)
                /home/vektah/go/src/github.com/external/tool (from $GOPATH)]
    packages_test.go:29: the package name should be set
FAIL
exit status 1
FAIL    github.com/vektah/packages-vendor-test  0.008s

This is hurting go modules adoption in http://github.com/99designs/gqlgen, Ideally I should be able to switch to packages to get modules support without losing support for vendor.

[mvdan]

I presume you're running the tool with GO111MODULE=on?

Also, see https://golang.org/cmd/go/#hdr-Maintaining_module_requirements; go build will not use vendor/ by default when ran in module mode, unlesss -mod=vendor, is given.

Also see #30240, which I believe would make this better and automatic in 1.13.

[vektah]

I presume you're running the tool with GO111MODULE=on?

No, I'm using GO111MODULE=auto from inside the gopath. The same failure happens when running GO111MODULE=off.

Note that in the test I am importing the vendored package directly to prove the go toolchain is happy to resolve this path, its only x/packages that cant find it.

[mvdan]

Interesting - I'm not sure why this would happen, as go/packages uses go list underneath. If you've vendored package foo/bar via vendor/foo/bar, what happens if you run go list -compiled foo/bar?

/cc @matloob @ianthehat

[vektah]

go list -compiled github.com/external/tool
can't load package: package github.com/external/tool: cannot find package "github.com/external/tool" in any of:
        /usr/local/go/src/github.com/external/tool (from $GOROOT)
        /home/adam/go/src/github.com/external/tool (from $GOPATH)

interestingly this is ok

go list -compiled github.com/vektah/packages-vendor-test/vendor/github.com/external/tool
github.com/vektah/packages-vendor-test/vendor/github.com/external/tool

which is consistent with go build

go build github.com/external/tool
can't load package: package github.com/external/tool: cannot find package "github.com/external/tool" in any of:
        /usr/local/go/src/github.com/external/tool (from $GOROOT)
        /home/adam/go/src/github.com/external/tool (from $GOPATH)

go build github.com/vektah/packages-vendor-test/vendor/github.com/external/tool
(OK)

[mvdan]

If I add a main.go file importing github.com/external/tool, go list -deps does the right thing:

$ go list -deps -f {{.Dir}}
/home/mvdan/go/src/github.com/vektah/packages-vendor-test/vendor/github.com/external/tool
/home/mvdan/tip/src/internal/cpu
/home/mvdan/tip/src/unsafe
/home/mvdan/tip/src/internal/bytealg
/home/mvdan/tip/src/runtime/internal/atomic
/home/mvdan/tip/src/runtime/internal/sys
/home/mvdan/tip/src/runtime/internal/math
/home/mvdan/tip/src/runtime
/home/mvdan/go/src/github.com/vektah/packages-vendor-test

So I really don't know where the bug is. Seems to me like go list works fine.

[mvdan]

Ah, I think I'm finally catching up - you can't do packages.Load(cfg, "github.com/external/tool") because go list github.com/external/tool doesn't work.

I think go/packages, like go list -deps, will look in the vendor folder to find dependencies, but not the direct packages you've asked it to load.

From https://golang.org/cmd/go/#hdr-Vendor_Directories:

Go 1.6 includes support for using local copies of external dependencies to satisfy imports of those dependencies, often referred to as vendoring.

So it seems to me like this is by design; vendor is only meant to satisfy dependencies.

Perhaps this will all work better in module mode with the new vendor folder; I'm not sure. /cc @bcmills

[vektah]

I think that dates back to a bunch of fixes to prevent tools that didn't know about vendoring from accidentally running across all of the vendored code. Hiding it from go list makes sense for the same reason.

I don't think this logic should be applied to x/packages, its a new module conceived well after vendoring was added, and its purpose is very different. x/loader used to check vendor, and x/packages checks the module path for dependencies, vendor search seems like a large omision here.

At the very least it should have a flag that mirrors ImportMode IgnoreVendor in build.

[matloob]

Hello @vektah, this is working as intended.

go/packages defers to the build system's (in this case, go build's and go list's) understanding of patterns you supply to it. So in your case, go/packages understands packages as go list does.

When run in GOPATH mode, go list's output is independent of the directory it's run in (with one exception below). And the way go list is specified in GOPATH mode, there's no way to supply which package you're "importing" a package from. Because we can't specify the importer of a package, go list has no way of knowing which vendor directories to consult to resolve a package.

The exception is that you can specify relative and absolute directory paths to go list. And as you noticed you can provide the qualified package paths pointing into the vendor directories. But it doesn't sound like either of those would solve your problem.

If you're trying to find a vendored package by its import path, your best bet is to do a deps query of an importer and look through those deps to find the package that you need.

[vektah]

this is working as intended.

Ok, that's a little disappointing. What is the intended use case for packages? Maybe I've misunderstood this to be a replacement for x/loader, because x/loader was not going to receive go modules updates?

I think not having a viable successor to X/loader is a big contributer to #24661

If you're trying to find a vendored package by its import path, your best bet is to do a deps query of an importer and look through those deps to find the package that you need.

I can't really do that, the code doesn't exist yet (this is for a codegen tool). I guess I could generate a new file in the right dir with a a bunch of _ imports, but that seems like a lot of work.

I might end up having to reproduce the is vendoring check and call go build directly for package resolution, before handing off to packages.

[matloob]

We do plan for go/packages to be a viable successor to loader, but it can't have all the features loader had because we want it both to be more correct and to be compatible with different build systems.

I'd like to know more about your use case to see if there's some other way for your tool to get the information it needs through the go/packages interface. I'm especially interested in how the vendored package name is provided to go/packages (by the user, by a codegen comment, is it created by the code generator?), and what the vendored package's relation is to the generated code.

If you'd like, I'd be able to meet via VC to discuss.

[vektah]

Its a code generator that takes some graphql schema and config that maps it to go types to bridge between the two type systems.

There are a heap of builtin graphql types like Int and String that are implemented in gqlgen, so when searching for those types it needs to find them in vendor. There are also often things like UUIDs and api types that are implemented in external libraries and referenced like this.

Its complicated, but if that isn't really clear I'm happy to jump on a call.

[matloob]

Are those types vendored into gqlgen itself? If so, can you use the "fully-qualified" package path to load the package? If not, where is it vendored?

[vektah]

Are those types vendored into gqlgen itself? If so, can you use the "fully-qualified" package path to load the package? If not, where is it vendored?

Those ones are, but they arent all defined by gqlgen, gqlgen is just adding its default implementations to config. End users can also refer directly to other external dependencies, eg well known protobuffer types and these full import paths shouldnt be leaking out.

For now we are now converting the standard import path from users to these full paths by calling build.Import before passing it off to x/packages.

[matloob]

Okay, I think this is a case of "working as intended; unfortunate". go list doesn't provide a way to look up that information, so we can't support it in go/packages. I think the workaround is reasonable for now, until everyone moves over to modules...

[seankhliao]

Closing as working as intended then