proposal: cmd/go: maintain 'classic' vendor behaviour

[kungfusheep]

Proposal: Maintain 'classic' vendor behaviour

This is a proposal to keep resolution of 'vendored' code consistent with the behaviour possible in go1.16 < versions of the compiler.

The express of intent to remove GOPATH from the compiler toolchain as soon as go1.17, and furthermore the GO111MODULE flag, combined with the fact -mod=vendor ignores vendor directories in locations other than the main module's root directory; means applications designed and written in the years before go mod could no longer compile.

To be clear, this proposal does not make an argument to keep GOPATH code resolution 1:1, but to allow code to be resolved from both the project root's vendor directory and any vendor directories found to be relative to import statements within.

In order to take any subsequent security updates and language features, teams are faced with non-insignificant engineering and logistical tasks to make this possible. For us and I imagine many others this is multiplied across a number of applications into the hundreds.

Our specific set up is to use git submodules of packages into the vendor directory, which is sufficient for our use-case of a large body of internally owned packages.

The applications have grown and adapted these structures over time based on the features available in the compiler until now, and the fact that until recently there was no 1st party solution for package management.

Removing the ability to resolve code in this way feels inconsistent with the spirit of the below paragraph taken from the go1compat document, which I appreciate people are likely familiar with.

It is intended that programs written to the Go 1 specification will continue to compile and run correctly, unchanged, over the lifetime of that specification. At some indefinite point, a Go 2 specification may arise, but until that time, Go programs that work today should continue to work even as future "point" releases of Go 1 arise (Go 1.1, Go 1.2, etc.).

Example

❯ go1.16 build
go: cannot find main module, but found .git/config in /Users/username/code/go/src/myapp
        to create a module there, run:
        go mod init

❯ go mod init
go: creating new go.mod: module myapp

❯ go1.16 build
vendor/git.lb.local/pkg/example/example.go:17:2: cannot find package "." in:
        /Users/username/code/go/src/myapp/vendor/github.com/protobuf/proto

In the above example, github.com/protobuf sits in the folder myapp/vendor/git.lb.local/pkg/example/vendor but is unable to be resolved. In go 1.16 this can be mitigated via the GO111MODULE flag.

References

• "We plan to drop support for GOPATH mode in Go 1.17"
• "Vendoring"
• "go1compat"

[tv42]

To continue using your git submodule layout while using modules, I think you can add a replace line to go.mod for each of the modules that reside in there, with a relative path starting with ./. Might be less confusing to combine that with a move from vendor/ to third_party/.

[rsc]

Unfortunately, "classic" vendor lets you end up with multiple copies of different (or even identical) minor versions of a package in the same build, which causes all manner of headaches. Adding that to module mode would introduce a special case that violates one of module mode's basic guarantees.

The applications have grown and adapted these structures over time based on the features available in the compiler until now, and the fact that until recently there was no 1st party solution for package management.

I apologize for the work you'll need to do to deduplicate your vendor directories into a single top-level vendor directory. All I can say is that if you make each one use a go.mod instead of vendor, that will set you up much better for understanding which code versions you are actually using, for updating in response to security fixes, and so on. It really does lead to a better, more agile setup.

I would push back a little on the "until recently" part. Go modules were first made available for use in Go 1.11 (August 2018), and we announced our intent to remove GOPATH in the "Go Modules in 2019" blog post (December 2018). Also, Go 1.16 will be supported until the release of Go 1.18 (~February 2022).

Removing the ability to resolve code in this way feels inconsistent with the spirit of the below paragraph taken from the go1compat document, which I appreciate people are likely familiar with.

Note also that the document explicitly excludes tools such as the Go command. It is concerned with the Go language and the standard library.

We do try to follow the spirit of the document generally, and again we are aware that it is painful to update old code trees. We don't intend to make a break like this again any time soon, if ever. But this one really is necessary for the progress of the Go ecosystem.

[ssxcxS45XT]

@rsc You statement assumes that people always uses other dependencies or such dependences always have version problems. But these simply are not always true. When those cases don't occur, it is unnecesary to enforce all Go code must be in a module. For this reason, the classical behavior should be maintained.