Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: document that vendor directories are ignored for packages without import paths #18007

Open
cespare opened this issue Nov 21, 2016 · 13 comments

Comments

Projects
None yet
9 participants
@cespare
Copy link
Contributor

commented Nov 21, 2016

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

Go 1.7.3 and tip

What operating system and processor architecture are you using (go env)?

linux/amd64

What did you do?

A package inside a directory named testdata cannot use vendored packages from above the testdata directory. Here's a demo:

$ export GOPATH=/tmp/gopath
$ mkdir -p $GOPATH/src/vendor/v
$ mkdir -p $GOPATH/src/a
$ cat > $GOPATH/src/vendor/v/v.go
package v

const V = "vendored"
$ cat > $GOPATH/src/a/a.go
package main

import "fmt"
import "v"

func main() { fmt.Println(v.V) }
$ cd $GOPATH/src/a
$ go build && ./a
vendored
$ mkdir -p $GOPATH/src/testdata
$ cp -r $GOPATH/src/a $GOPATH/src/testdata/
$ cd $GOPATH/src/testdata/a
$ go build
a.go:4:8: cannot find package "v" in any of:
        /Users/caleb/apps/go/src/v (from $GOROOT)
        /tmp/gopath/src/v (from $GOPATH)

This came up while writing tests for a program that itself invokes go build to build and run a temp binary.

I'm aware that testdata is special. It's documented in go help test:

The go tool will ignore a directory named "testdata", making it available to hold ancillary data needed by the tests.

and go help packages:

Directory and file names that begin with "." or "_" are ignored by the go tool, as are directories named "testdata".

but it wasn't obvious to me that not being able to use vendored packages is a consequence of this. Should it have been? Is this intended?

@bradfitz bradfitz added this to the Go1.9 milestone Nov 21, 2016

@bradfitz

This comment has been minimized.

Copy link
Member

commented Nov 21, 2016

We'll figure out whether it was intentional during the Go 1.9 cycle It's not Go 1.8 material since it's presumably been like this for a few cycles. Correct me if that's wrong.

@pascaldekloe

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2016

More specifically, Go fails to apply the vendor directory for subdirectories of testdata. That is, $GOPATH/src/testdata/*.go can use $GOPATH/src/testdata/vendor.

@pascaldekloe

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2016

Go 1.8 rejects testdata entirely. (see also cl31665 and #17597)

unexpected directory layout:
	import path: github.com/pascaldekloe/goe/verify
	root: /Users/pdekloe/repo/colfer/src
	dir: /Users/pdekloe/repo/colfer/src/github.com/pascaldekloe/colfer/testdata/vendor/github.com/pascaldekloe/goe/verify
	expand root: /Users/pdekloe/repo/colfer/src
	expand dir: /Users/pdekloe/repo/colfer/src/github.com/pascaldekloe/colfer/testdata/vendor/github.com/pascaldekloe/goe/verify
	separator: /

pascaldekloe added a commit to pascaldekloe/colfer that referenced this issue Dec 10, 2016

Move generated code out of testdata and into a separate directory per…
… language.

Mixing C++ and Go source files fail with: "can't build package github.com/pascaldekloe/colfer/testdata because it contains C++ files (Colfer.cpp) but it's not using cgo nor SWIG"

Go 1.8 fails to compile source code in "testdata". See also: golang/go#18007
@cespare

This comment has been minimized.

Copy link
Contributor Author

commented Feb 27, 2017

I'm not sure what the previous comment means, but when I run my repro in Go 1.8 I get exactly the same output as I did in 1.7.3.

@rsc rsc modified the milestones: Go1.10, Go1.9 Jun 12, 2017

@meilihao

This comment has been minimized.

Copy link

commented Jun 28, 2017

I get the same in go1.8 and go1.9beta2 linux/amd64:

chen@chen-pc:~/git/go/src/text$ go build
unexpected directory layout:
	import path: github.com/go-sql-driver/mysql
	root: /home/chen/git/go/src
	dir: /home/chen/git/go/src/text/vendor/github.com/go-sql-driver/mysql
	expand root: /home/chen/git/go/src
	expand dir: /home/chen/git/go/src/text/vendor/github.com/go-sql-driver/mysql
	separator: /

But my project is text:

chen@chen-pc:~/git/go/src/text$ ll -a
总用量 16
drwxr-xr-x  3 chen chen 4096 6月  28 22:57 .
drwxr-xr-x 12 chen chen 4096 6月  28 22:55 ..
-rwxr-xr-x  1 chen chen 1031 6月  28 22:30 main.go
drwxr-xr-x  4 chen chen 4096 6月  28 22:46 vendor
chen@chen-pc:~/git/go/src/text$ echo $GOPATH 
/home/chen/git/go
chen@chen-pc:~/git/go/src/text$ pwd
/home/chen/git/go/src/text
@pascaldekloe

This comment has been minimized.

Copy link
Contributor

commented Jun 28, 2017

@meilihao

This comment has been minimized.

Copy link

commented Jun 28, 2017

@pascaldekloe Thinks for your answer.

@rsc rsc modified the milestones: Go1.10, Go1.11 Dec 1, 2017

@mvdan

This comment has been minimized.

Copy link
Member

commented Jan 17, 2018

I've just hit this myself too, and was confused by how this limitation isn't documented anywhere.

Like @cespare, I was using testdata as a GOPATH, with my/pkg/testdata/src/foo depending on bar (available at my/pkg/vendor/bar).

I currently don't see an easy workaround. I could make the test make a full copy of vendor/* into my/pkg/testdata/src/*, but that's a bit clunky. I think that is enough reason to lift this restriction, unless there is a good reason for it to stay there.

@bcmills

This comment has been minimized.

Copy link
Member

commented Jan 18, 2019

vendor does not work outside of GOPATH in GOPATH mode (see #14566).
This use of testdata is arguably outside of the GOPATH import tree, since the package doesn't have an addressable import path relative to GOPATH/src.

Given that, I don't think this issue will be fixed. Closing as a dup of #14566.

@bcmills bcmills closed this Jan 18, 2019

@cespare

This comment has been minimized.

Copy link
Contributor Author

commented Jan 18, 2019

Hey @bcmills, thanks for taking a look. However, I don't follow the argument you're making.

This use of testdata is arguably outside of the GOPATH import tree, since the package doesn't have an addressable import path relative to GOPATH/src.

Can you explain this more? Packages inside a testdata directory are treated as though they're in the surrounding GOPATH today. You can import any package you like from the GOPATH, just like packages outside testdata, except that you cannot import from vendor. You can even import packages that transitively use vendor packages and that works fine.

I've run into this multiple times with tests that build Go code. In one case we had a test that worked fine for months but then started failing because the generated code it was building started (directly) using a vendored package.

If code in testdata "isn't in GOPATH", then it should behave like that and it should be documented. It should not behave as though it is in GOPATH but with one tiny difference.

@cespare

This comment has been minimized.

Copy link
Contributor Author

commented Jan 22, 2019

Given that it remains surprising and undocumented, I'm reopening this issue. I understand that the resolution may not be the fix I'm asking for (make vendor work inside testdata) but the behavior should at least make sense in a way that's derivable from the documentation.

@cespare cespare reopened this Jan 22, 2019

@bcmills

This comment has been minimized.

Copy link
Member

commented Jan 23, 2019

Can you explain this more? Packages inside a testdata directory are treated as though they're in the surrounding GOPATH today. You can import any package you like from the GOPATH, just like packages outside testdata, except that you cannot import from vendor.

Packages inside testdata only halfway exist in GOPATH mode. Sometimes they have pseudo-import-paths with leading underscores, and sometimes they aren't visible at all. The fact that you can import them at all from outside of the testdata directory is almost certainly a bug.

_gopath/src$ mkdir -p ./example.com/foo/testdata/bar

_gopath/src$ mkdir -p ./example.com/foo/testdata/baz

_gopath/src$ cat >./example.com/foo/foo.go <<EOF
package foo

import _ "example.com/foo/testdata/bar"
EOF

_gopath/src$ cat >./example.com/foo/testdata/bar/bar.go <<EOF
package bar
EOF

_gopath/src$ cat >./example.com/foo/testdata/baz/baz.go <<EOF
package baz

import (
        _ "example.com/foo"
        _ "example.com/foo/testdata/bar"
)
EOF

_gopath/src$ export GO111MODULE=off

_gopath/src$ go list example.com/...
example.com/foo

_gopath/src$ go list -deps example.com/foo
example.com/foo/testdata/bar
example.com/foo

_gopath/src$ go list -deps example.com/foo/testdata/baz
example.com/foo/testdata/bar
example.com/foo
example.com/foo/testdata/baz

_gopath/src$ cd example.com/foo/testdata

_gopath/src/example.com/foo/testdata$ go list ./...
_/tmp/tmp.BkLPsJ6gxB/_gopath/src/example.com/foo/testdata/bar
_/tmp/tmp.BkLPsJ6gxB/_gopath/src/example.com/foo/testdata/baz

_gopath/src/example.com/foo/testdata$ go list -deps ./...
_/tmp/tmp.BkLPsJ6gxB/_gopath/src/example.com/foo/testdata/bar
example.com/foo/testdata/bar
example.com/foo
_/tmp/tmp.BkLPsJ6gxB/_gopath/src/example.com/foo/testdata/baz

@bcmills bcmills changed the title cmd/go: packages inside testdata dir cannot see vendored packages cmd/go: document that vendor directories are ignored for packages without import paths Jan 23, 2019

@bcmills bcmills modified the milestones: Unplanned, Go1.13 Jan 23, 2019

@bcmills

This comment has been minimized.

Copy link
Member

commented Jan 23, 2019

the behavior should at least make sense in a way that's derivable from the documentation.

Retitled accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.