-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: cmd/go: go mod: limit version resolution to packages that are consumed #52296
Comments
cc @liggitt @thockin (our veteran go module wranglers) @thaJeztah I love this! frankly ANYTHING other than status quo would be very welcome change. Thanks for taking the time and drafting this as a proposal |
not to detract from the main point of the issue, but kubernetes/kubernetes does not use |
This sounds like it would make version resolution dramatically slower (everything from go get/list/tidy), requiring downloading and scanning the source code for every potential dependency revision. |
I guess that's something to look into, if it's possible to optimize. (As mentioned, I'm not deeply familiar with who all version resolution is performed); I can imagine this being mostly an issue when adding / removing / updating the list of dependencies in Perhaps there's other solutions possible, and I'm definitely open to alternatives. I opened this proposal because there's a real issue that quite some projects that I'm (directly/indirectly) involved in struggle with this on a daily base, and I'd like to see if we can improve the status quo, without such projects digging themselves in further with hacks and workarounds trying to fix issues with go modules. |
😅 perhaps it was a bad example; even then, I wonder if kubernetes would've needed the "please, don't use kubernetes/kubernetes as a module" situation (where modules are extracted from the main repository) if things were different? Also, if (e.g.) (but going way off-topic now 😂) |
What you are describing is very similar in both motivation and effect to the module graph pruning added in Go 1.17 (proposed in #36460). Since most modules in the wild have not yet upgraded their |
To my understanding, module graph pruning helps with transitive dependencies, but not with the issues described in this ticket (I may be wrong though), which is why I was hoping this could trigger a conversation on how to solve this. |
Ah, I see. You actually do depend on Right, so module graph pruning doesn't help there, but there also isn't an inexpensive alternative. If we restricted the analysis to the specific packages imported by your module, we would have to load the entire package import graph just to know which dependencies to use. With the Go 1.17 module graph pruning, we don't need to do that — all of the upfront analysis is based purely on (It took a fair amount of thinking for us to come up with a design with that property! It's important for efficiency, and the graph-pruning invariants are designed very carefully to support it — it isn't something we can discard lightly. 😅) |
So, in general you have two options. One is to avoid upgrading The other is to apply
would notch out the transitive dependency from IMO |
Thanks for taking the time to write up those options (very much appreciated!)
I appreciate "performance". That said (as I mentioned above); version resolution only is needed when updating a dependency. It's a "one-time" performance penalty. On projects where a single run of CI can take hours (or literally days in total compute time), I think that's acceptable. The "logrus" and "containerd" situation is of course "fictive". In projects with a large dependency tree, things are usually more complicated (a more practical example could be These issues often can arise from some indirect dependency deep in the dependency tree. More often than not, some small module that's (e.g.) following "best practice", and enables Now, these things can be solved in "my project" (say, In the ideal situation, things "explode", and their code fails to build, helping them notify "something's wrong!". Often, things are more subtle than that, and consumers may be living on a ticking timebomb (not theoretical; this happened on more than one occasion, sometimes even resulting in security issues). If I realise this is a tough problem to solve, and (to some extend), go modules may not have been designed for large projects, but it's the situation we're in, so trying to find what options are possible. As mentioned; some of these problems can be "solved" or hacked/worked around by large projects themselves, but these projects also have a responsibility towards their respective consumers (of which there may be many), so I'm also trying to explore if there's a solution that would address that side. |
I thought it ran on pretty much every |
When lazy module loading is in effect (that is, if the main module is at So, pretty much every |
Thx! Yes, so this is what I had in mind when I wrote that (but I'm sure there's gonna be situations I missed, where the performance impact can be an issue - let's see what/if we can come to something that works and satisfies most use-cases). |
This proposal has been added to the active column of the proposals project |
Thanks @rsc ! FWIW, if you or the other maintainers think it may be useful to have a synchronous call with some project maintainers to brainstorm, let me know: happy to reach out to them to make that happen. |
Hi @bcmills
I think this might be implied by your last parenthetical, but just to make it explicit -- my understanding is that would not help the original example here? In other words, for the FWIW, I put together a quick example here, including to possibly help the conversation with something more concrete: https://github.com/thepudds/test-go-mod-52296-a/releases/tag/v0.3.0 The There might be multiple reasons for that, but I suspect part of it might be (from go mod ref):
... where the client of foobar has loaded the go.mod of containerd and seen the logrus v1.8.1 requirement in the v1.6.2 containerd go.mod, and hence ends up with the "bad" logrus v1.8.1. And related, the client of foobar has a
Sorry if any of this is off base. Finally, that's not to say |
Hi @thaJeztah
I suspect that is not quite right. FWIW, I also put together another hypothetical with a different package-level import graph (compared to your original example above): https://github.com/thepudds/test-go-mod-52296-a/releases/tag/v0.4.0 In short:
In other words, it is an example where There are more details in that link, including an additional example showing how the exact sequence of events can influence whether or not
I think it is correct that multiple excludes can be needed, which is what I did here. |
That is correct, and it is an intentional property of the design. (See the section titled Upgrade Speed in the original MVS blog post.) As a counterbalance, though, note that you don't have to upgrade that tiny dependency in your own module until you are ready. |
In principle it would be possible to load only the parts of the module graph corresponding to the main module's import graph, and that could indeed avoid a few unwanted upgrades. That's a benefit. But we would need to trade that benefit against the costs, and the costs as I see them are numerous:
I just don't see the cost/benefit tradeoff working out — those are significant costs, and the marginal benefit of putting off upgrades of dependencies just doesn't seem that high in comparison, especially given the available workarounds. (If your project is large enough to have a lot of dependencies, you'll probably have to upgrade them eventually anyway!) |
Based on the discussion above, this proposal seems like a likely decline. |
No change in consensus, so declined. |
@thaJeztah wrote::
FYI, if anyone is still following this issue, there is some related conversation in #48429 (comment) and in other comments in that "cmd/go: track tool dependencies in go.mod" proposal. |
go modules resolve the minimum required version of dependencies based on the
go.mod
of modules used by a project. This works well for small modules where the list of dependencies ingo.mod
is representative for the code, but is problematic for larger modules that provide many packages.Let's illustrate with an example.
Example: project "foobar"
This is our "foobar" project. It uses logrus to print "Hello foobar":
Project foobar requires logrus 1.7.0 - it can't currently use a newer version of this dependency, because it has change in behavior that causes foobar to break (of course, SemVer should guard us against breaking changes, but the world isn't perfect, so we specify we want v1.7.0):
No project would be complete without an AppArmor check, and containerd provides an implementation for this. It's a small package, and
pkg/apparmor
has no dependencies, other than Go stdlib (apparmor.go, apparmor_linux.go, apparmor_unsupported.go).So we add the containerd v1.6.2 dependency:
However, checking our
go.mod
;Adding containerd as a dependency forced us to also updates the
logrus
dependency to a newer (for us "incompatible") version (as well as updates thegolang.org/x/sys
dependency), even though none of the files in containerd'spkg/apparmor
package use this dependencyWhile the project "foobar" example is of course just to illustrate the problem, this issue is problematic for many real-life situations (some more details below).
Current "solutions"
There are various "solutions" for this problem, but they're not for the faint of heart.
A. Use
replace
rulesProjects can add a
replace
rule to force go modules to use a fixed version. While this helps "us" (the "foobar project" maintainers) build and ship our project, it's a different story for consumers of the "foobar" module;replace
rules are not transitional, and because of this, all projects depending on our module will (out of the box) be "forced" to use the newer version, unless they copy the replace rules.Various (sometimes "high profile") projects currently use
replace
rules (read them, and weep! 😭😭😭), e.g.: containerd and kubernetes. Worst of all, using replace rules (especially when use to the extend as the kubernetes example) throws out one of the biggest advantages of go modules; version resolution / management.B. Separate modules (multi-module repository)
We can ask the
containerd
maintainers to providepkg/apparmor
as a separate module. While this may be an option in some cases, maintaining a multi-module repository gets complicated fast;replace
rules may be needed to make sure code it tested against the version in the repository (not the latest released version of the module)In short; unless "you're Google", or have a dedicated team of engineers to set up automation to perform these actions (e.g., the complicated release procedures for the kubernetes project), maintaining a multi-module repository is complicated, and in many a heavy burden for project maintainers.
C. Separate modules (multiple repositories)
We can ask the
containerd
maintainers to providepkg/apparmor
as a separate module in a separate repository.While this gives a clearer separation between the modules, it shares the same (if not more) problems as the previous solution. Maintaining a separate repository can add significant overhead for project maintainers (and in some cases may be restricted due to (company) policies). In addition, not all packages may be suitable to become a module / project of their own (let's not encourage creating another "leftpad").
D. Just copy the code! (It's open source, y'all!)
Unfortunately, this solution has been chosen on many occasions. I don't think this needs explaining why this should not be a preferred solution.
What did you expect to see? (proposed solution)
I'd like to see go modules to only consider version resolution based on the packages that are actually consumed from a module. Go modules conflates all packages in a repository, resulting in the (main) go module /
go.mod
to become a collection of all possible dependencies that may be needed (depending on which packages are consumed from the module).While go modules won't use dependencies if they're not used by any code, version resolution is still be influenced by them (see the example above). I'm not very familiar with the internals of go module's tooling, but I think go has all the "building blocks" available to make this possible;
It's able to provide which imports a package needs:
With that information, it could;
go.mod
And, if in future containerd's
pkg/apparmor
would introduce a new dependency, that's the moment it gets its "right to vote" in the version-resolution for that dependency.The text was updated successfully, but these errors were encountered: