Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: {get|build|install} need to improve behaviors with vendor directory. #44841

Open
james-lawrence opened this issue Mar 7, 2021 · 14 comments
Open

Comments

@james-lawrence
Copy link
Contributor

@james-lawrence james-lawrence commented Mar 7, 2021

What version of Go are you using (go version)?

go version
go version go1.16 linux/amd64

Does this issue reproduce with the latest release?

yes

this is more of an experience report vs a specific bug/proposal around things I've found frustrating with go tooling since moving to modules for the last year.

vendor setups are treated as second class citizens.

this shows itself in a number of ways.

  1. in a repository that contains vendor directory dependencies go get doesn't automatically install to the vendor directory. requiring an additional command.
  2. accesses the network to fetch dependencies when they're in the vendor directory.
  3. numerous claims by go developers that go modules will allow for removing vendor support. this is not true and will never be true. in theory yes, you can build identical programs from go.mod and related infrastructure. however it ignores things like: reliance of 3rd party infrastructure or having to spinning up your own infrastructure to do so. vendoring is essentially that infrastructure in a single folder. it use to be far easier to use, understand, maintain, audit. however it has always been treated as as second class since support was added to satisfy the community at large.

go get updates dependencies without explicit user action.

I've run into this a number of types where I'm working run so go tooling commands and when I build/run into an error where my vendor is out of sync with go.mod only to find a bunch of modules have been updated. Unfortunately I don't know exact which command is causing this, I suspect go get or go mod tidy but its just a gut feeling based on what I remember doing each time i've run into this. what I do know is this has occurred to me multiple times now where dependencies I was not interacting with in any way have been updated.

possibly related:

  • #40053 which spawned this thread which i'm just now seeing because github never notified me on the updates to the issue.

go modules has caused an explosion in complexity in the go environment

it has almost doubled the number of environment variable flags. its responsible for 9 out of 20 environment variables.

  • GOPROXY
  • GONOPROXY
  • GOPRIVATE
  • GONOSUMDB
  • GOSUMDB
  • GOVCS
  • GOINSECURE
  • GOMODCACHE

most of which are accessing the network and all of which could essentially be dropped from a developers consciousness with a functional vendor directory. in fact if a vendor directory is detected in the main module during get/install/build you could in fact treat it as the gomodcache unless upgrading or adding a new package (in which case you're interacting with the VCS to populate that cache).

@kevinburkemeter
Copy link

@kevinburkemeter kevinburkemeter commented Mar 7, 2021

Basically the way I work around this is by providing an extra command that sets all of the flags appropriately, and then just asking everyone to run make vendor.

vendor:
	envdir envs/localdev go mod vendor && envdir envs/localdev go mod tidy

And then envs/localdev contains

$ ll envs/localdev | grep GO
-rw-r--r--  1 kevin staff   3 2019-08-14 16:21 GO111MODULE
-rw-r--r--  1 kevin staff  33 2019-10-04 12:12 GOFLAGS
-rw-r--r--  1 kevin staff 209 2020-01-16 21:49 GONOSUMDB
-rw-r--r--  1 kevin staff  19 2020-01-16 21:49 GOPRIVATE
-rw-r--r--  1 kevin staff   4 2020-01-16 21:49 GOSUMDB

in particular GOFLAGS is -mod=vendor -race, GO111MODULE set to on, and the other ones disable the public checksum db and cache.

@kevinburkemeter
Copy link

@kevinburkemeter kevinburkemeter commented Mar 7, 2021

I'm not saying this is a good solution, it took a while to get there, but having it has simplified working with the vendor directory pretty well.

@james-lawrence
Copy link
Contributor Author

@james-lawrence james-lawrence commented Mar 7, 2021

@kevinburkemeter appreciate the workaround, looking for the discussion on how we as a community can fix the tool so it can provide a great experience out of the box without twiddling bits and bobs for the basic use cases. there is also a fundamental tooling problem underlying all of these issues which i don't think has been addressed @rsc touches on it in that thread I linked.
moving to a offline/read only starting point for gomodcaches could potentially resolve a ton of this complexity.

@oiooj oiooj added the GoCommand label Mar 8, 2021
@toothrot toothrot changed the title go {get|build|install} need to improve behaviors with vendor directory. cmd/go: {get|build|install} need to improve behaviors with vendor directory. Mar 8, 2021
@toothrot
Copy link
Contributor

@toothrot toothrot commented Mar 8, 2021

@toothrot toothrot added this to the Backlog milestone Mar 8, 2021
@bcmills
Copy link
Member

@bcmills bcmills commented Mar 8, 2021

  1. in a repository that contains vendor directory dependencies go get doesn't automatically install to the vendor directory. requiring an additional command.

That was an intentional decision resulting from the discussion in #30240 (see #30240 (comment) for a summary); see also #29058 and #38222.

The final design as implemented was #33848.

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 8, 2021

go get updates dependencies without explicit user action.

The go command's default behavior was changed in Go 1.16 (see proposal #42466). The go.mod file is no longer updated implicitly by default except in go get and go mod tidy. However, it was already not updated implicitly when -mod=vendor is set.

Note that go get and go mod tidy do continue to update the go.mod file — and do not support the -mod flag at all. That is because the primary purpose of those commands is to update the go.mod file: they are “explicit user action”. (If you want to install a binary without changing your dependencies, use go install instead of go get.)

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 8, 2021

  1. accesses the network to fetch dependencies when they're in the vendor directory.

That should never be the case when -mod=vendor is set (implicitly or explicitly).

When vendoring is in use, the only times the go command should access the network are when you run an explicit go get command (which has the primary purpose of accessing the network to update your dependencies) or an explicit go mod subcommand (which has the primary purpose of either resolving or reporting information that is not present in the vendor directory).

If you find that a go command other than go get or go mod is accessing the network when -mod=vendor is set (implicitly or explicitly), please file a separate issue with steps to reproduce it.

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 8, 2021

  1. numerous claims by go developers that go modules will allow for removing vendor support.

As far as I am aware, no one on the Go project is claiming that modules will allow for removing vendor support. (If we were planning to remove it, we would not have bothered to implement #33848!)

Per the Background section on #33848:

Go users have built a variety of workflows around vendored dependencies for a variety of use-cases — including self-contained builds, language-agnostic code review, and ephemeral patches — and since the launch of module mode in Go 1.11 they have made it clear that those workflows remain important in module mode.

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 8, 2021

in fact if a vendor directory is detected in the main module during get/install/build you could in fact treat it as the gomodcache unless upgrading or adding a new package (in which case you're interacting with the VCS to populate that cache).

Sorry, but that is not feasible. The vendor directory contains individual packages, not complete modules. It also does not in general include the go.mod files needed to reconstruct the dependencies of the modules containing those packages.

However, note that with -mod=vendor today the go command already does use the vendor directory to load packages and dependency information “unless upgrading or adding a new package”. Also note that -mod=vendor has been the default behavior when a vendor directory is present since Go 1.14. (See https://golang.org/doc/go1.14#vendor.)

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 9, 2021

Regarding the environment variables:

GOPROXY
GOSUMDB

These variables do admittedly add complexity. However, GOPROXY improves the availability of (non-vendored) dependencies, such as for new projects using existing libraries, and GOSUMDB improves security and reproducibility by ensuring that everyone agrees on the exact contents of the versions they require. We believe that those benefits substantially outweigh the cost of the added complexity.

GONOPROXY
GOPRIVATE
GONOSUMDB

GOPRIVATE sets the defaults for GONOPROXY and GONOSUMDB. The latter are finer-grained controls, but most users should only need to bother with GOPRIVATE, which allows users to trade off the security and availability benefits mentioned above against the benefit of being able to work with code that is not publicly available.

GOMODCACHE

The default value for GOMODCACHE is derived from GOPATH — this is another one of those variables that provides a finer-grained control for something that most users will probably set via a coarse-grained

GOINSECURE

GOINSECURE replaces the -insecure flag, which is deprecated (and planned for removal in Go 1.17) because it is too insecure (#37519). So this one isn't adding complexity so much as replacing existing complexity, and it comes with a substantial increase in security.

GOVCS

GOVCS is mostly unnecessary for public dependencies, because the public module proxy currently allows all of the supported VCS tools (with sandboxing to mitigate the security risks of less-well-tested tools).

(GOVCS is a bit like GOINSECURE, in that it allows for finer-grained security policies when updating private dependencies — improving security by default, but still allowing an opt-out for private dependencies on specific trusted hosts.)

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 9, 2021

this is more of an experience report vs a specific bug/proposal

As far as I can tell, many of the points here were either addressed, or explicitly considered and rejected, in #33848.

In order to revisit the behaviors rejected for #33848, we would need to see new information (something not already considered in the design and decisions for that proposal, such as a failure mode or mode of use that we were not aware of at the time), and probably also a concrete proposal of what to do differently.

@gudvinr

This comment was marked as off-topic.

@bcmills

This comment was marked as off-topic.

@james-lawrence
Copy link
Contributor Author

@james-lawrence james-lawrence commented Mar 29, 2021

@bcmills i'll get around to responding hopefully next weekend. but i think the problem with refering to #33848 is i believe the go team it making fundamental mistakes w/ respect to go modules that make it harder to work with golang tooling and defeating some of the main benefits to the tooling. the last few releases I've run into more and more problems with the tooling than at anytime in the 4 years prior.

and the problems start with how packages are resolved (and particularly the lack of a library for dealing with them), and progress into how all these commands interact with modules.

basically my intuition is telling me that the go team is taking a remote first approach to the resolution vs a local filesystem first approach. which is the source of most of the ergonomics of problems with vendoring. (-mod=vendor is a hack and unintuitive solution and dare I say it entirely unnecessary) and with the replace directive.

i'm particularly frustrated with the fact other attempts at addressing specific problems have been rejected out of hand because the goteam thinks they've considered the problem and 'solved' them. usually by saying 'we'll deal with it late if it comes up' and when it does come up 'we've already considered and rejected that its problem' despite the fact it is a problem and sometimes regressions from previous behavior.

I believe I have some ideas to addressing the issues without causing any major issues going forward. but means the go team needs to take a step back and listen vs decreeing that they've 'already considered the issue'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants