Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: cannot install vendored binary in module aware mode #45115

Open
bhcleek opened this issue Mar 18, 2021 · 6 comments
Open

cmd/go: cannot install vendored binary in module aware mode #45115

bhcleek opened this issue Mar 18, 2021 · 6 comments

Comments

@bhcleek
Copy link
Contributor

@bhcleek bhcleek commented Mar 18, 2021

What version of Go are you using (go version)?

$ go version
go version go1.16.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN="/Users/bhcleek/go/bin"
GOCACHE="/Users/bhcleek/Library/Caches/go-build"
GOENV="/Users/bhcleek/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/bhcleek/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/bhcleek/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.2"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/f9/d6g779t53gq_20w0zm2f_1zm0000gn/T/go-build1429675000=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

go install $MODULE/vendor/some/vendored/command in module aware mode.

What did you expect to see?

The command to be installed as it is in GOPATH mode.

What did you see instead?

cannot find package "." in:
        /Users/bhcleek/src/$MODULE/vendor/some/vendored/command
@cherrymui cherrymui changed the title cannot install vendored binary in module aware mode cmd/go: cannot install vendored binary in module aware mode Mar 19, 2021
@cherrymui cherrymui added this to the Backlog milestone Mar 19, 2021
@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Mar 19, 2021

This may be working as intended. But it would be good if someone who is more familiar with modules could confirm or correct me. cc @bcmills

If it is intended, perhaps the error could be made clearer.

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 19, 2021

@bhcleek, is the source code for the command actually present in /Users/bhcleek/src/$MODULE/vendor/some/vendored/command?

(Can you provide specific steps to reproduce the problem, including the steps used to populate the vendor directory?)

@bhcleek
Copy link
Contributor Author

@bhcleek bhcleek commented Mar 19, 2021

Yes, it's actually present.

Here's a script to duplicate this issue. Notice that it does go install in both modes to demonstrate the differences:

#!/bin/bash

set -eu -o pipefail

MODULE="github.com/golang/issue-45115"
cd $(mktemp -d)
export GOPATH=$PWD

mkdir -p src/$MODULE
cd src/$MODULE
printf "=====> $PWD\n" >&2

go mod init github.com/golang/issue-45115

cat <<EOF > anchor.go
// +build tools

package foo

import (
	_ "github.com/digitalocean/gta/cmd/gta"
)
EOF

go get github.com/digitalocean/gta/cmd/gta@latest
go mod tidy
go mod vendor

printf "=====> installing in GOPATH mode\n" >&2

GO111MODULE=off go install github.com/golang/issue-45115/vendor/github.com/digitalocean/gta/cmd/gta

printf "=====> install in module aware mode\n" >&2

GO111MODULE=on go install github.com/golang/issue-45115/vendor/github.com/digitalocean/gta/cmd/gta
@bcmills
Copy link
Member

@bcmills bcmills commented Mar 19, 2021

Thanks.

github.com/golang/issue-45115/vendor/github.com/digitalocean/gta/cmd/gta is a package import path, not a filesystem path.

In GOPATH mode, each vendored copy of a package has its own unique identity, in order to distinguish it from all of the other (distinct) copies of github.com/digitalocean/gta/cmd/gta that might be vendored into the tree.

In contrast, in module mode with -mod=vendor, the vendored dependencies of the main module are authoritative: there is only one copy of github.com/digitalocean/gta/cmd/gta and it is the one in ./vendor/github.com/digitalocean/gta/cmd/gta. So it does not receive a mangled or expanded import path; it is just github.com/digitalocean/gta/cmd/gta, full stop, and it is the only such package that will be used during a build.

So in module mode, the command you're looking for should either use the actual canonical import path:

$ GO111MODULE=on go install github.com/digitalocean/gta/cmd/gta

or, if you want to be agnostic to whether you are in module mode, you can use the absolute filesystem path (which is recognizably not a package import path):

$ GO111MODULE=on go install $(pwd)/vendor/github.com/digitalocean/gta/cmd/gta

go list confirms that those two paths are identical in module mode, but distinct in GOPATH mode:

$ GO111MODULE=on go list -f '{{.ImportPath}}: {{.Dir}}'  github.com/digitalocean/gta/cmd/gta
github.com/digitalocean/gta/cmd/gta: /tmp/tmp.1Kc7aaRkbU/.gopath/src/github.com/golang/issue-45115/vendor/github.com/digitalocean/gta/cmd/gta

$ GO111MODULE=on go list -f '{{.ImportPath}}: {{.Dir}}' $(pwd)/vendor/github.com/digitalocean/gta/cmd/gta
github.com/digitalocean/gta/cmd/gta: /tmp/tmp.1Kc7aaRkbU/.gopath/src/github.com/golang/issue-45115/vendor/github.com/digitalocean/gta/cmd/gta

$ GO111MODULE=off go list -f '{{.ImportPath}}: {{.Dir}}' $(pwd)/vendor/github.com/digitalocean/gta/cmd/gta
github.com/golang/issue-45115/vendor/github.com/digitalocean/gta/cmd/gta: /tmp/tmp.1Kc7aaRkbU/.gopath/src/github.com/golang/issue-45115/vendor/github.com/digitalocean/gta/cmd/gta
@bcmills
Copy link
Member

@bcmills bcmills commented Mar 19, 2021

I think if anything this is a documentation issue, but I'm not sure which documentation we would need to update.

@stevetraut: do our new modules user-docs have a section on how to create and use vendored dependencies (go mod vendor and -mod=vendor)?

@bhcleek
Copy link
Contributor Author

@bhcleek bhcleek commented Mar 30, 2021

@bcmills I think this may be more than a documentation issue. Today I found some inconsistencies in how vendored commands can be built in module aware mode.

A vendored golang.org/x/tools/cmd/stringer will install with or without the vendor directory in the import path path without error in module aware mode. Both of these seem to work when stringer is vendored:

GO111MODULE=on go install do/vendor/golang.org/x/tools/cmd/stringer
GO111MODULE=on go install golang.org/x/tools/cmd/stringer

Stringer is not in a module, so it would seem that there are differences in go install behavior when the vendored tool is in a module vs not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants