Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: modules: derive implicit exclusions from requirements #49274

Open
misberner opened this issue Nov 1, 2021 · 5 comments
Open

Comments

@misberner
Copy link

misberner commented Nov 1, 2021

Currently, the go module (dependency) graph contains all transitive dependencies, including from versions which do not end up being the effectively resolved version. While this is unavoidable to some extent (determining the effective version requires traversing and unfolding the graph), the same does not apply for the root requirements: if a root requirement for example.com/my/module v1.2.3 exists, we know that a strictly transitive requirement example.com/my/module v1.2.2 (or any lower version) cannot be a candidate for the finally resolved version. Furthermore, this determination can be made without having to inspect any additional (i.e., other than the main module's one) go.mod files. A similar (but less practically relevant) observation applies to the main module: any transitive requirement for example.com/main/module v1.2.3 can be ignored, as the main module with version "" always wins.

This proposal therefore aims at changing the module dependency resolution algorithm in such a way that every requirement for the main module of any version, as well as any strictly transitive requirement for any other module of a version less than the version directly required by the main module, is treated as if an exclude directive for that module/version combination existed.

This is motivated by the following:

  • reduce the clutter in go.sum
  • reduce the number of "orphaned" module dependencies, i.e., modules that show up in go list -m all, even though they are not (transitively) required by any module in their respective selected version.
  • reduce the amount of manual replace or exclude hackery required in case a project's dependencies module hygiene is suboptimal.

While all of the above problems don't get eliminated by this proposal (they can basically all still exist one level down in the dependency graph/DAG), they should get significantly mitigated. In contrast to more sophisticated heuristics to address these, this proposal is limited to relying on information that is already present in the go.mod file, but not spelled out in replace directives.

This would require compatibility logic for go mod tidy, as the resulting go.sum and even go.mod file output might change, as well as the requirements for go.sum entries that may effect the module dependency resolution throughout.

@misberner
Copy link
Author

misberner commented Nov 1, 2021

(sorry, I forgot to apply the Proposal label on creation, and it looks like I can't do this afterwards)

@ianlancetaylor ianlancetaylor changed the title porposal: cmd/go: modules: derive implicit exclusions from requirements proposal: cmd/go: modules: derive implicit exclusions from requirements Nov 2, 2021
@gopherbot gopherbot added this to the Proposal milestone Nov 2, 2021
@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) Nov 2, 2021
@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Nov 2, 2021

CC @bcmills @matloob

@ianlancetaylor ianlancetaylor added the GoCommand cmd/go label Nov 2, 2021
@gopherbot

This comment has been minimized.

@hanchaoqun

This comment has been minimized.

@bcmills
Copy link
Member

bcmills commented Dec 10, 2021

if a root requirement for example.com/my/module v1.2.3 exists, we know that a strictly transitive requirement example.com/my/module v1.2.2 (or any lower version) cannot be a candidate for the finally resolved version

Unfortunately, some users (and third-party tools) do make direct edits to the go.mod file, so in generally we cannot assume that a requirement necessarily reflects a conscious decision to override some other requirement in the module graph.

However, as of Go 1.17 we are able to prune out the dependencies of a lot of lower-than-selected versions of modules; see https://go.dev/ref/mod#graph-pruning.

The problems motivating this proposal seem to me to be essentially the same problems that motivated graph pruning. Specifically:

  • Graph pruning does reduce the clutter in go.sum, or at least can do so eventually. (A go 1.17 module still has a large go.sum to ease migration from the 1.16 toolchain, but a go 1.18 module should generally have a much smaller go.sum, especially as its dependencies move to go 1.17 or higher.)
  • Graph pruning does reduce “orphaned” module dependencies. (Those are exactly the dependencies that are “pruned out” of the module graph.)
  • Graph pruning also reduces (but does not completely eliminate) the need for replace or exclude workarounds for bad transitive dependencies, because it ends up pruning out many irrelevant requirements on incompatible modules.

Given graph pruning, I don't think this proposal would be worth its implementation complexity; at the very least, I think we need a couple of release cycles to see how well graph pruning works out in practice before we consider pursuing changes aimed at closely-related problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Incoming
Development

No branches or pull requests

5 participants