Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: emit an explicit error when packages in GOPATH/src/vendor overlap with GOROOT/src/vendor #34068

Open
javasgl opened this issue Sep 4, 2019 · 19 comments
Assignees
Labels
Milestone

Comments

@javasgl
Copy link

@javasgl javasgl commented Sep 4, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13 darwin/amd64

Does this issue reproduce with the latest release?

only in go1.13 happened, back to go1.12.9 can fix this

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/username/Library/Caches/go-build"
GOENV="/Users/username/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH=""
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/b1/bhf9j3q96m75223l3hh41ccw0000gn/T/go-build173249759=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

go install -v ./...

What did you expect to see?

no errrors

What did you see instead?

src/vendor/golang.org/x/net/http2/frame.go:17:2: use of vendored package not allowed
src/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:28:2: use of vendored package not allowed
src/vendor/golang.org/x/net/http2/transport.go:35:2: use of vendored package not allowed

@javasgl javasgl changed the title Breaking changes! use of vendored package not allowed 1.13 Breaking changes! use of vendored package not allowed Sep 4, 2019
@javasgl javasgl changed the title 1.13 Breaking changes! use of vendored package not allowed go1.13 Breaking changes! use of vendored package not allowed Sep 4, 2019
@mvdan

This comment has been minimized.

Copy link
Member

@mvdan mvdan commented Sep 4, 2019

Please provide full instructions to show that this worked on 1.12, and broke in 1.13. For example, using the official Docker images.

@mvdan mvdan added the WaitingForInfo label Sep 4, 2019
@bcmills bcmills changed the title go1.13 Breaking changes! use of vendored package not allowed cmd/go: "use of vendored package not allowed" Sep 4, 2019
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 4, 2019

@javasgl, to amplify what @mvdan said: go install -v ./... doesn't help us reproduce your problem unless you also include the contents of the module or repo that you were working in.

In particular, do the import statements in your code include the vendor/ prefix?

This change probably relates to CL 164619, but we did add a regression test in that CL to try to avoid any change in behavior for existing code using the correct import paths.

@bcmills bcmills added this to the Go1.14 milestone Sep 4, 2019
@psanford

This comment has been minimized.

Copy link

@psanford psanford commented Sep 4, 2019

I think I ran into the same issue. I made an example repo here to demonstrate the issue: https://github.com/psanford/gobug34068

If you checkout that repo and set your GOPATH to the git working directory you can try building package example.com/foo from go 1.12 (works) vs 1.13 (errors). example.com/foo is a grpc client hello world example.

Grpc and its dependencies were vendored in src/vendor via gvt.

@javasgl

This comment has been minimized.

Copy link
Author

@javasgl javasgl commented Sep 5, 2019

@bcmills same project structure with @psanford

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 12, 2019

Found a much smaller reproducer.

-- issue34068/issue34068.go --
package issue34068

import (
	_ "net/http"
	_ "golang.org/x/net/http2/hpack"
)
-- $GOPATH/src/vendor/golang.org/x/net/http2/hpack/hpack.go --
package hpack
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 12, 2019

I looked into this today, but I'm not sure that it's feasible to fix.

The trouble is that a number of caches within cmd/go/internal/load assume (and have long assumed) that every package has a unique canonical import path, and import paths whose first component does not contain a dot are in general reserved for the standard library.

The simple failure mode is this:

  1. First we load golang.org/x/net/http2/hpack from within GOPATH, and resolve it to the path vendor/golang.org/x/net/http2/hpack with location GOPATH/src/vendor/golang.org/x/net/http2/hpack.
    2 Then we load net/http from within GOROOT, and we also resolve it to vendor/golang.org/x/net/http2/hpack. The cache shows the location of that package to be GOPATH/src/vendor/golang.org/x/net/http2/hpack, but if we loaded it from scratch we would have gotten GOROOT/src/vendor/golang.org/x/net/http2/hpack.

A similar failure mode occurs with the relative ordering of GOPATH and GOROOT swapped.

A long-term fix would be to have all of the caches track packages by directory instead of import path. That change is fairly invasive, and in general we are not pursuing long-term fixes in GOPATH mode (although we do try to avoid regressions).

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 12, 2019

Fortunately, there are (at least) two workarounds available:

  1. You can move the vendor directory from the top level (GOPATH/src/vendor) to one level in (GOPATH/src/example.com/vendor). Then its path no longer conflicts with the vendor subdirectory in GOROOT.

  2. You can convert the project to use module mode instead of GOPATH mode, in which case the vendor prefix is no longer added to the canonical path for the packages contained in the project's vendor directory (because there is only one vendor replacement for any given package path).

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 12, 2019

Finally, I will note that this issue does not reproduce for any project compatible with go get. Since the vendor directory for any such project must be within its repository, go get would have placed that vendor directory within the projects subdirectory in GOPATH/src, not at the root of GOPATH/src.

@bcmills bcmills self-assigned this Sep 12, 2019
@bcmills bcmills changed the title cmd/go: "use of vendored package not allowed" cmd/go: "use of vendored package not allowed" when packages in GOPATH/src/vendor overlap with GOROOT/src/vendor Sep 12, 2019
@bronze1man

This comment has been minimized.

Copy link
Contributor

@bronze1man bronze1man commented Sep 18, 2019

Hi @bcmills ,
Here is my real problem that tigger this bug:

I have a git repository that use a submodule at GOPATH/src/vendor to store others' repository code, then I come across this bug.
As I need to compile my project with go1.9.7, so I can not translate my repository to use module mode (go version higher than go1.9 use a little more memory with my application, so it can not work in ios9 with extension process which limit the total memory to about 10MB).
I also try to write a program to symlink others' repository into GOPATH/src/xxx , but it do not work in windows 10 in go1.12 (I do not try go1.13 with this bug).

I think I have two solutions for this problem now for me to continue work with go1.13 and go1.9:

  • I can try move GOROOT/src/vendor/golang.org/x/net/http2/hpack into another directory in GOROOT , like GOROOT/src/internal/golang.org/x/net/http2/hpack, and replace all the import in the GOROOT. This solution should not break my code. But I need do that again when I upgrade to go1.14
  • Move all my code from GOPATH/src/xxx to GOPATH/src/a/xxx and move GOPATH/src/vendor to GOPATH/src/a/xxx. And replace all the import. But this solution will curse a lot of git diff in the import when I merge other branch of my repository.
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 18, 2019

I also try to write a program to symlink others' repository into GOPATH/src/xxx , but it do not work in windows 10 in go1.12 (I do not try go1.13 with this bug).

Generally the go command works reasonably well with file symlinks, but not with directory symlinks.
(As far as I am aware, my solution for symlinking testdata inputs in misc/cgo/test works on Windows, but I'm not sure whether that approach works with Go versions older than 1.13.)

You could also consider copying the files instead of symlinking them.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 18, 2019

I need to compile my project with go1.9.7

Given that Go 1.9 is no longer receiving security updates, I would recommend that you reconsider that approach. (Have you filed an issue for the increase in memory footprint? Can you rework other parts of the application to make it work within the desired footprint with a supported Go version?)

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 18, 2019

Finally, is there a reason you need to map in the submodule at GOPATH/src/vendor rather than GOPATH/src/$YOUR_DOMAIN/vendor? That should not change the import paths needed in the packages within $YOUR_DOMAIN.

@bronze1man

This comment has been minimized.

Copy link
Contributor

@bronze1man bronze1man commented Sep 19, 2019

Hi @bcmills ,

Generally the go command works reasonably well with file symlinks, but not with directory symlinks.

I tried to create directory symlinks in windows 10, so it do not work. Copying the files on windows 10 should be a good idea, but the build script and .gitignore file maybe another story.

Given that Go 1.9 is no longer receiving security updates, I would recommend that you reconsider that approach.

Another consideration of not using module mode is I do not know how to copy all the source code of dependencies of my project into one directory and then compile there right now.(looks like that is GOPATH mode.) The only way I can sure that my project can recompile success in a long time when I have all the dependencies in my git repository or in my own storage.(The network may fail, but my local disk will work in a long time.)

is there a reason you need to map in the submodule at GOPATH/src/vendor rather than GOPATH/src/$YOUR_DOMAIN/vendor?

I have a lots of packages in GOPATH/src/app/ep/main.go or GOPATH/src/make/main.go, they all need import same package in vendor, If I add vendor to GOPATH/src/config/vendor and GOPATH/src/app/ep/main.go. The binary size will increase and I will need symlinks or copy or vendor directories.

I have another workaround of this bug:

  • I can use 2 GOPATH, first one is my project repository, second one is my vendor repository. It should only change my build script and the idea configure.
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 27, 2019

I have a lots of packages in GOPATH/src/app/ep/main.go or GOPATH/src/make/main.go

Per https://golang.org/doc/code.html#ImportPaths:

For your own packages, you must choose a base path that is unlikely to collide with future additions to the standard library or other external libraries.

If you keep your code in a source repository somewhere, then you should use the root of that source repository as your base path. For instance, if you have a GitHub account at github.com/user, that should be your base path.

Package paths that do not start with a domain name are in general reserved for the standard library. That rules out dotless names like app and make, especially in module mode.

You should start your package paths with a domain or prefix that you control, and then you can root your vendor directory at that prefix too.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 27, 2019

Another consideration of not using module mode is I do not know how to copy all the source code of dependencies of my project into one directory and then compile there right now.

See https://golang.org/cmd/go/#hdr-Make_vendored_copy_of_dependencies and https://golang.org/cmd/go/#hdr-Modules_and_vendoring.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 27, 2019

Given the existence of workarounds, I think the way forward on this is to change the go command to explicitly reject collisions between GOPATH/src/vendor and GOROOT/src/vendor with a useful error message.

@bcmills bcmills changed the title cmd/go: "use of vendored package not allowed" when packages in GOPATH/src/vendor overlap with GOROOT/src/vendor cmd/go: error out when packages in GOPATH/src/vendor overlap with GOROOT/src/vendor Sep 27, 2019
@bcmills bcmills added the NeedsFix label Sep 27, 2019
@bcmills bcmills changed the title cmd/go: error out when packages in GOPATH/src/vendor overlap with GOROOT/src/vendor cmd/go: emit an explicit error when packages in GOPATH/src/vendor overlap with GOROOT/src/vendor Sep 27, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@bronze1man

This comment has been minimized.

Copy link
Contributor

@bronze1man bronze1man commented Oct 10, 2019

I can use 2 GOPATH, first one is my project repository, second one is my vendor repository. It should only change my build script and the idea configure.

  • I have to config my ide idea to support two gopath, as i have 10+ projects, this should be a lot of work for everyone in my team.(the ide config can not git sync to others' computers)
  • I have to change all my build script to support 2 GOPATH and add a config to my projects.

So here is my new workaround(which is less hard work for me right now):

  • Place my git submodule of golang vendor into GOPATH/src/vendor.
  • Modify golang source code, move the GOROOT/src/vendor to another place, and delete any check of not allow write code in GOPATH/src/vendor. And upload it to cdn and ask others to install this modify version of golang. This version of golang should work with all standard golang code, it should not break compatibility.

I think modify golang source code should be the best workaround for me now. It is I only need to modify once every golang release.

Here is the workaround bash script(in case me or someone need it again):

cd /usr/local/go
cp -rf ./src/vendor ./src/crypto/vendor
cp -rf ./src/vendor ./src/net/vendor
cp -rf ./src/vendor ./src/go/vendor
rm -rf ./src/vendor
cd ./src
./make.bash
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Oct 10, 2019

@bronze1man, forking the entire Go toolchain seems a lot more invasive than prefixing your import path with a domain name, or copying your vendor contents into GOPATH/src rather than mapping them in to GOPATH/src/vendor.

If the problem is an increase in binary size, have you filed an issue about that?

@bronze1man

This comment has been minimized.

Copy link
Contributor

@bronze1man bronze1man commented Oct 10, 2019

forking the entire Go toolchain seems a lot more invasive than prefixing your import path with a domain name,

I do not agree that, our team have 3 huge golang projects, each one have 100+ golang packages and 4000+ golang files, prefixing the import path into one a domain name means I need to rename almost all of the golang files. I do have a tool to do it automatic, but there will be a lot of path relative bugs to fix manually. That is a lot more invasive than modify and forking the entire Go toolchain.

By the way, I have modified the golang source code 1.13.1, and upload to our cdn , this workaround works great.

or copying your vendor contents into GOPATH/src rather than mapping them in to GOPATH/src/vendor.

  • This workaround works and it is easy, but I can not use git submodule to manage all of the vendor packages. It will became difficult to change the vendor packages across our 3 huge golang projects.I may need to develop another tool to auto copy and compare the different of those vendor packages(As there are 3 copys of them in different projects)

If the problem is an increase in binary size, have you filed an issue about that?

  • Binary size and memory footprint problem have been solved by use go1.9.7 right now. All golang version after 1.9 is worse.

It looks like the using of GOPATH/src/vendor has calcified within our team. If golang do not forbidden it in the begin (like version go1.0), then there will be a lot of usage of it, make it very hard to change.
I am sorry to tell you, I need to maintain a fork of golang source code from golang 1.13.

jpuhlman added a commit to MontaVista-OpenSourceTechnology/meta-virtualization that referenced this issue Dec 2, 2019
Source: meta-virtualization
MR: 00000
Type: Integration
Disposition: Merged from meta-virtualization
ChangeID: 00cc8afd1e80e73f1205691c4ce440498b52eba5
Description:

Since go was upgraded to 1.13, there is a failure:
...
| src/vendor/golang.org/x/net/http2/frame.go:17:2: use of vendored package not allowed
| ../../../recipe-sysroot/usr/lib64/go/src/net/http/h2_bundle.go:49:2: use of vendored package not allowed
...

Refer upstream suggestion [1]:
`or copying your vendor contents into GOPATH/src rather than
mapping them in to GOPATH/src/vendor.'

[1] golang/go#34068

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
Signed-off-by: Jeremy Puhlman <jpuhlman@mvista.com>
wr-linux-build-srv pushed a commit to WindRiver-OpenSourceLabs/meta-virtualization that referenced this issue Dec 2, 2019
Since go was upgraded to 1.13, there is a failure:
...
| src/vendor/golang.org/x/net/http2/frame.go:17:2: use of vendored package not allowed
| ../../../recipe-sysroot/usr/lib64/go/src/net/http/h2_bundle.go:49:2: use of vendored package not allowed
...

Refer upstream suggestion [1]:
`or copying your vendor contents into GOPATH/src rather than
mapping them in to GOPATH/src/vendor.'

[1] golang/go#34068

Issue: LINCD-465
(LOCAL REV: NOT UPSTREAM) -- sent to meta-virtualization on 20191201

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
armcc pushed a commit to lgirdk/meta-virtualization that referenced this issue Dec 2, 2019
Since go was upgraded to 1.13, there is a failure:
...
| src/vendor/golang.org/x/net/http2/frame.go:17:2: use of vendored package not allowed
| ../../../recipe-sysroot/usr/lib64/go/src/net/http/h2_bundle.go:49:2: use of vendored package not allowed
...

Refer upstream suggestion [1]:
`or copying your vendor contents into GOPATH/src rather than
mapping them in to GOPATH/src/vendor.'

[1] golang/go#34068

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
wr-linux-build-srv pushed a commit to WindRiver-OpenSourceLabs/meta-virtualization that referenced this issue Dec 3, 2019
Since go was upgraded to 1.13, there is a failure:
...
| src/vendor/golang.org/x/net/http2/frame.go:17:2: use of vendored package not allowed
| ../../../recipe-sysroot/usr/lib64/go/src/net/http/h2_bundle.go:49:2: use of vendored package not allowed
...

Refer upstream suggestion [1]:
`or copying your vendor contents into GOPATH/src rather than
mapping them in to GOPATH/src/vendor.'

[1] golang/go#34068

Issue: LINCD-465
(LOCAL REV: NOT UPSTREAM) -- sent to meta-virtualization on 20191201

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.