Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: vendored dependencies lack language version annotations #36876

Open
bcmills opened this issue Jan 29, 2020 · 11 comments
Open

cmd/go: vendored dependencies lack language version annotations #36876

bcmills opened this issue Jan 29, 2020 · 11 comments
Labels
Milestone

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 29, 2020

In writing #36875, I realized that we will have a bit of a problem with vendored dependencies if and when we start making incompatible changes to the language.

Specifically, the vendor directory is a flattened package tree, and it does not include go.mod files unless the vendored package dependencies happen to include a package at the root of the vendored module. Many modules (such as golang.org/x/tools) don't even have an importable package at the root, let alone one that someone would want to import.

Unfortunately, the go.mod files are currently the only place where we record the language version to use for those dependencies. (I don't know what language version we end up using for -mod=vendor builds today, but I don't see how it could possibly be the correct one in general.)

I can think of two possible fixes:

  1. Extend the ## annotation comments that we added for #33848 so that, for any module that provides one or more packages, we annotate the go version (if any) found in the dependency's original go.mod file. (Fortunately, we intentionally designed that part of the format to tolerate the addition of new annotations.)
    For example:

    # golang.org/x/text v0.3.3
    ## explicit, go 1.14
    golang.org/x/text/number
    
  2. Explicitly copy in the go.mod files for each module that provides one or more packages, even if there is no corresponding package in that part of the vendor tree.

I have a slight preference for approach (1) but don't feel strongly about it.

(CC @ianlancetaylor @griesemer @jayconrod @matloob @mvdan)

@mvdan
Copy link
Member

@mvdan mvdan commented Jan 29, 2020

If anyone upgrades to a new Go version, and updates their go.mod to use that newer language version which happens to remove a language feature, could the build break until the author re-runs go mod vendor? If that's planned, we should make sure to mention it well in the changelog.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Jan 29, 2020

@mvdan, ideally, a user changing the language version in their own go.mod file should not affect the language version used to compile their dependencies (regardless of vendoring), and therefore should not require any update to their vendor tree.

@mvdan
Copy link
Member

@mvdan mvdan commented Jan 29, 2020

Hm, maybe I misunderstood. I thought re-running go mod vendor with Go 1.15 could add a bunch of those extra ## lines, necessary to add the missing language version information for vendored dependencies.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Jan 29, 2020

Re-running go mod vendor with Go 1.15 presumably would add some ## lines, but I don't think that should be conditioned on the go version declared in the main module's go.mod file.

To minimize churn in vendor/modules.txt, we could omit the version annotations for any module that specifies a version at least as old as the default (#36875). As long as those dependencies built with Go 1.14 (or whatever default version we choose), they should continue to build without re-vendoring.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Jan 29, 2020

(The go version annotations probably do not need a consistency check like the one added in #33848, because they currently have no effect on the computed module dependency graph.)

@thepudds
Copy link

@thepudds thepudds commented Jan 29, 2020

@bcmills

  1. Explicitly copy in the go.mod files for each module that provides one or more packages, even if there is no corresponding package in that part of the vendor tree.

FWIW, I would suggest thinking a bit more about the possible virtues of your approach 2., including as things evolve in the future (e.g., perhaps another bit of information might get added to go.mod in the future).

Also, some small-ish chance that storing the current go.mod files might be a stepping stone towards storing old go.mod files as well, which might end up enabling more cmd/go functionality when in vendor mode (e.g., #30240 (comment) and next couple comments there), though I think storing the old go.mod files was not a preferred approach in the past.

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Jan 30, 2020

I'd lean toward option (1). Seems simpler.

go.mod files affect module boundaries, so it would be hard to get this right. @bcmills mentioned in a meeting that commands run inside the vendor directory would get confused (a go.mod inside the vendor directory would define the main module, possibly the wrong main module). Tools and editors would have this problem, too.

@bcmills bcmills added the NeedsFix label Feb 5, 2020
@gopherbot gopherbot removed the NeedsDecision label Feb 5, 2020
@bcmills
Copy link
Member Author

@bcmills bcmills commented Feb 5, 2020

Seems like the vendor/modules.txt annotation is the way to go: that way we only have to load one file (vendor/modules.txt) to be able to build packages in vendor mode, rather than N different go.mod files, and the diffs are generally smaller, and we won't have to change the version logic if we happen to make go mod vendor start pruning out go.mod files (for example, in order to be less confusing for tools and editors).

@andybons
Copy link
Member

@andybons andybons commented Apr 16, 2020

@bcmills what are the next steps here?

@bcmills
Copy link
Member Author

@bcmills bcmills commented Apr 18, 2020

Since there haven't been any breaking language changes in Go 1.15, this doesn't need to be a release-blocker. (I do still plan to add the annotation, but I'm not sure whether it will make 1.15.)

@bcmills bcmills modified the milestones: Go1.15, Go1.16 May 4, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 24, 2020

Change https://golang.org/cl/244078 mentions this issue: cmd/go/internal/modload: cache the Go language version for each module globally

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.