Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: support vendoring in workspace mode #60056

Open
matloob opened this issue May 8, 2023 · 3 comments
Open

proposal: cmd/go: support vendoring in workspace mode #60056

matloob opened this issue May 8, 2023 · 3 comments
Milestone

Comments

@matloob
Copy link
Contributor

matloob commented May 8, 2023

This is a proposal to support vendoring in workspace mode.

We originally didn't support vendoring of a workspace to simplify the implementation of workspaces, but there's recently been demand by users of vendoring who want to workspaces for that support, so now is a good time to consider adding that support in.

go work vendor subcommand

Under this proposal, a new subcommand, go work vendor, will be added to the go command. go work vendor will check to make sure that workspace mode is active when it starts, (and return an error if it isn't) and if it is, it will write a vendor directory for the workspace. The vendor directory will be created in the same directory as the go.work file. The vendor directory will contain packages taken from the modules in the build list of the workspace. vendor/modules.txt will contain the set of packages from each of the modules in the build list as usual, will list modules with # explicit annotations for each module required by one of the workspace modules, and will annotate replacements for each of the effective replacements in the workspace (That is, if the go.work file overrides replacements in workspace only the replacement in the go.work file will be annotated.)

go work vendor otherwise behaves the same as and takes the same flags as go mod vendor.

go mod vendor subcommand

go mod vendor will explicitly fail if a go.work file is found and GOWORK is not set to off. See discussion below.

-mod=vendor in workspace mode.

Under this proposal, when workspace mode is active, if -mod=vendor is set explicitly, or it is set implicitly through the existence of a vendor directory in the same directory as the go.work The go command will enter workspace mode and interpret the vendor directory as reflecting the workspace's modules. The go command will use the vendor directory in the same directory as go.work. vendor/modules.txt is interpreted as reflecting the modules in the workspace. That interpretation is done through the vendoring consistency check, and the structure of the internal module graph built.

The vendoring consistency check would check vendor/modules.txt against the workspace modules. It will check that all explicitly required modules appear in vendor/modules.txt with an # explicit annotation and all modules in vendor/modules.txt with # explicit are required by a workspace module. Similarly, all effective replacements in the workspace are expected to appear in vendor/modules.txt and all replacements in vendor/modules.txt are expected to appear in a workspace module go.mod or the go.work file.

The way the module graph is built is similar to to that of an unpruned module. The module graph contains the main modules as its roots, and each of the main modules would require their module requirements in addition to a fake vendor/modules.txt module that in turn requires the set of modules in the vendorList (the set of modules in vendor/modules.txt` that provide packages.

Otherwise, vendor mode with a workspace behaves the same as vendor mode with a module. The packages in the vendor directory are used to satisfy imports in the main modules when loading packages and doing builds.

Other considerations.

go work vendor and go mod vendor.

Under the proposal as it is, go mod vendor works to vendor the module it's in, but fails if there's a go.work file present and GOWORK is not set to off. If that wasn't the case, there could be confusion if a go.mod and go.work files exist in the same directory. Then go mod vendor and go work vendor would create vendor directories in the same place, but with different contents. Since in workspace mode, the vendor directory is interpreted as a reflecting the workspace, if go mod vendor is run before another command in workspace mode, the vendor consistency checks will likely fail. When they fail the error message will give the user the appropriate command to run go mod vendor or go work vendor depending on whether modules.txt was checked against a single module or a the workspace. Even so, the user might still get confused about why running go mod vendor and immediately running another command might result in an error. To reduce the confusion, we require a user to not have a go.work file present or have GOWORK set to off when go mod vendor is run, so that when a following command is run under the same conditions, it doesn't interpret the vendor directory as reflecting the workspace.

The behavior might break already existing workflows where go mod vendor is run when a go.work is present and module vendoring is being used, but it seems better to do so than to introduce the potential confusion of the user creating a vendor directory that fails consistency checks.

@cherrymui cherrymui added GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels May 8, 2023
@cherrymui cherrymui added this to the Backlog milestone May 8, 2023
@thockin
Copy link

thockin commented May 8, 2023

The lack of vendor support makes go workspaces very difficult for Kubernetes to adopt. I don't care if it is exactly identical to classic vendor support, but we need something...

  • We actually review all changes to pinned deps.
  • K8s has been burned more than once by upstream repos which disappear, force-push and obliterate history, or change their vanity go-get URLs to non-code things to intentionally break people.
  • I have personally been burned by trying to work on an airplane with what I THOUGHT was a full set of downloaded modules, but it turns out it was not really the full set.
  • The public Go module proxy does not retain every SHA forever.
  • The protoc runs we do require access to gogo.proto, which is currently satisfied via vendor/.../gogo.proto

@gopherbot
Copy link

Change https://go.dev/cl/495801 mentions this issue: cmd/go: add support for vendoring in workspace mode

@matloob matloob changed the title cmd/go: support vendoring in workspace mode proposal: cmd/go: support vendoring in workspace mode May 31, 2023
@matloob matloob modified the milestones: Backlog, Proposal May 31, 2023
@matloob matloob removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 1, 2023
@gopherbot
Copy link

Change https://go.dev/cl/499977 mentions this issue: gopls/internal/regtest/codelens: set GOWORK=off for go mod vendor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants