proposal: cmd/go: go mod: limit version resolution to packages that are consumed

[thaJeztah]

go modules resolve the minimum required version of dependencies based on the go.mod of modules used by a project. This works well for small modules where the list of dependencies in go.mod is representative for the code, but is problematic for larger modules that provide many packages.

Let's illustrate with an example.

Example: project "foobar"

This is our "foobar" project. It uses logrus to print "Hello foobar":

mkdir foobar && cd foobar

cat > main.go <<EOF
package main

import (
    "github.com/sirupsen/logrus"
)

func main() {
    logrus.Info("Hello foobar")
}
EOF

go mod init foobar

Project foobar requires logrus 1.7.0 - it can't currently use a newer version of this dependency, because it has change in behavior that causes foobar to break (of course, SemVer should guard us against breaking changes, but the world isn't perfect, so we specify we want v1.7.0):

go mod edit -require github.com/sirupsen/logrus@v1.7.0
go mod tidy

cat go.mod
module foobar

go 1.18

require github.com/sirupsen/logrus v1.7.0

require golang.org/x/sys v0.0.0-20191026070338-33540a1f6037 // indirect

No project would be complete without an AppArmor check, and containerd provides an implementation for this. It's a small package, and pkg/apparmor has no dependencies, other than Go stdlib (apparmor.go, apparmor_linux.go, apparmor_unsupported.go).

cat > main.go <<EOF
package main

import (
    "github.com/containerd/containerd/pkg/apparmor"
    "github.com/sirupsen/logrus"
)

func main() {
    if apparmor.HostSupports() {
        logrus.Infof("Running Foobar Deluxe, with AppArmor")
    } else {
        logrus.Infof("Running Foobar Basic")
    }
}
EOF

So we add the containerd v1.6.2 dependency:

go mod edit -require github.com/containerd/containerd@v1.6.2
go mod tidy

However, checking our go.mod;

Adding containerd as a dependency forced us to also updates the logrus dependency to a newer (for us "incompatible") version (as well as updates the golang.org/x/sys dependency), even though none of the files in containerd's pkg/apparmor package use this dependency

$ cat go.mod
module foobar

go 1.18

require github.com/sirupsen/logrus v1.8.1

require (
	github.com/containerd/containerd v1.6.2
	golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e // indirect
)

While the project "foobar" example is of course just to illustrate the problem, this issue is problematic for many real-life situations (some more details below).

Current "solutions"

There are various "solutions" for this problem, but they're not for the faint of heart.

A. Use replace rules

Projects can add a replace rule to force go modules to use a fixed version. While this helps "us" (the "foobar project" maintainers) build and ship our project, it's a different story for consumers of the "foobar" module; replace rules are not transitional, and because of this, all projects depending on our module will (out of the box) be "forced" to use the newer version, unless they copy the replace rules.

Various (sometimes "high profile") projects currently use replace rules (read them, and weep! 😭😭😭), e.g.: containerd and kubernetes. Worst of all, using replace rules (especially when use to the extend as the kubernetes example) throws out one of the biggest advantages of go modules; version resolution / management.

B. Separate modules (multi-module repository)

We can ask the containerd maintainers to provide pkg/apparmor as a separate module. While this may be an option in some cases, maintaining a multi-module repository gets complicated fast;

• modules become separate entities (need to be tested separately)
• if there's dependencies between packages (now separate modules), replace rules may be needed to make sure code it tested against the version in the repository (not the latest released version of the module)
• if these modules are expected to be used externally, each of them has to be tagged/released separately
• which, in combination with "inter-module dependencies" also means tagging and releasing MUST be performed in the correct order (to make sure all modules use the latest release)

In short; unless "you're Google", or have a dedicated team of engineers to set up automation to perform these actions (e.g., the complicated release procedures for the kubernetes project), maintaining a multi-module repository is complicated, and in many a heavy burden for project maintainers.

C. Separate modules (multiple repositories)

We can ask the containerd maintainers to provide pkg/apparmor as a separate module in a separate repository.

While this gives a clearer separation between the modules, it shares the same (if not more) problems as the previous solution. Maintaining a separate repository can add significant overhead for project maintainers (and in some cases may be restricted due to (company) policies). In addition, not all packages may be suitable to become a module / project of their own (let's not encourage creating another "leftpad").

D. Just copy the code! (It's open source, y'all!)

Unfortunately, this solution has been chosen on many occasions. I don't think this needs explaining why this should not be a preferred solution.

What did you expect to see? (proposed solution)

I'd like to see go modules to only consider version resolution based on the packages that are actually consumed from a module. Go modules conflates all packages in a repository, resulting in the (main) go module / go.mod to become a collection of all possible dependencies that may be needed (depending on which packages are consumed from the module).

While go modules won't use dependencies if they're not used by any code, version resolution is still be influenced by them (see the example above). I'm not very familiar with the internals of go module's tooling, but I think go has all the "building blocks" available to make this possible;

It's able to provide which imports a package needs:

go list -json ./pkg/apparmor/ | jq .Imports
[
  "os",
  "sync"
]

With that information, it could;

• take all dependencies listed in the module's go.mod
• remove all direct dependencies that are not used by the packages that are consumed
• use the remaining dependencies to perform version resolution

And, if in future containerd's pkg/apparmor would introduce a new dependency, that's the moment it gets its "right to vote" in the version-resolution for that dependency.

[thaJeztah]

/cc @dims @kzys @dmcgowan @tonistiigi @cpuguy83 @tianon (had this idea as a "draft" / "note" let me know if this idea makes sense to you 😅)

[dims]

cc @liggitt @thockin (our veteran go module wranglers)

@thaJeztah I love this! frankly ANYTHING other than status quo would be very welcome change. Thanks for taking the time and drafting this as a proposal

[liggitt]

Various (sometimes "high profile") projects currently use replace rules (read them, and weep! 😭😭😭), e.g.: containerd and kubernetes. Worst of all, using replace rules (especially when use to the extend as the kubernetes example) throws out one of the biggest advantages of go modules; version resolution / management.

not to detract from the main point of the issue, but kubernetes/kubernetes does not use replace directives to force downlevel versions. This is enforced in presubmit, and there are no replace directives that select versions other than the require versions in the submodules published out of kubernetes that are consumed as libraries (for example, client-go).

[seankhliao]

This sounds like it would make version resolution dramatically slower (everything from go get/list/tidy), requiring downloading and scanning the source code for every potential dependency revision.

[thaJeztah]

This sounds like it would make version resolution dramatically slower (everything from go get/list/tidy), requiring downloading and scanning the source code for every potential dependency revision.

I guess that's something to look into, if it's possible to optimize. (As mentioned, I'm not deeply familiar with who all version resolution is performed); I can imagine this being mostly an issue when adding / removing / updating the list of dependencies in go.mod. After that's done, the resolved versions are "fixated", so a go get of the main module would already have performed all resolution.

Perhaps there's other solutions possible, and I'm definitely open to alternatives. I opened this proposal because there's a real issue that quite some projects that I'm (directly/indirectly) involved in struggle with this on a daily base, and I'd like to see if we can improve the status quo, without such projects digging themselves in further with hacks and workarounds trying to fix issues with go modules.

[thaJeztah]

there are no replace directives that select versions other than the require versions in the submodules published out of kubernetes that are consumed as libraries (for example, client-go).

😅 perhaps it was a bad example; even then, I wonder if kubernetes would've needed the "please, don't use kubernetes/kubernetes as a module" situation (where modules are extracted from the main repository) if things were different? Also, if (e.g.) go mod (tidy|vendor) --recursive and/or tooling was available to manage multi-module repositories (updating inter-module versions), to not having to script all of that.

(but going way off-topic now 😂)

[bcmills]

What you are describing is very similar in both motivation and effect to the module graph pruning added in Go 1.17 (proposed in #36460).

Since most modules in the wild have not yet upgraded their go.mod files to list go 1.17 or higher, I think it's premature to propose follow-on changes on top of that — we should get more modules upgraded to go 1.17 first (so that pruning can be more effective), and only then evaluate whether there is more that we can usefully do to trim down the dependency graph.

[thaJeztah]

To my understanding, module graph pruning helps with transitive dependencies, but not with the issues described in this ticket (I may be wrong though), which is why I was hoping this could trigger a conversation on how to solve this.

[bcmills]

Ah, I see. You actually do depend on containerd, and that module has a direct dependency on logrus.

Right, so module graph pruning doesn't help there, but there also isn't an inexpensive alternative. If we restricted the analysis to the specific packages imported by your module, we would have to load the entire package import graph just to know which dependencies to use. With the Go 1.17 module graph pruning, we don't need to do that — all of the upfront analysis is based purely on go.mod files, and there are typically far fewer of those than .go source files for packages.

(It took a fair amount of thinking for us to come up with a design with that property! It's important for efficiency, and the graph-pruning invariants are designed very carefully to support it — it isn't something we can discard lightly. 😅)

[bcmills]

So, in general you have two options.

One is to avoid upgrading containerd until you have identified and fixed the logrus incompatibility (either by upstreaming a fix to logrus itself, or by upstreaming a fix to whatever other dependency is not compatible with the current logrus release).

The other is to apply exclude (not replace!) directives to notch out the dependencies that you know to be spurious. For example:

exclude github.com/sirupsen/logrus v1.8.1

would notch out the transitive dependency from containerd, essentially asserting “yeah, it's required by my dependencies but I know it isn't relevant”. Due to module graph pruning, your own exclude directive would also prune out that dependency for users who depend on your module, although of course they could still end up with it in their own dependency graphs (for example, if they end up transitively importing containerd).

IMO exclude is best used only temporarily, to buy you some time for the aforementioned upstream fixes. At some point, there isn't really a viable alternative to maintaining compatibility with the latest releases of your dependencies.

[thaJeztah]

Thanks for taking the time to write up those options (very much appreciated!)

Right, so module graph pruning doesn't help there, but there also isn't an inexpensive alternative

I appreciate "performance". That said (as I mentioned above); version resolution only is needed when updating a dependency. It's a "one-time" performance penalty. On projects where a single run of CI can take hours (or literally days in total compute time), I think that's acceptable.

The "logrus" and "containerd" situation is of course "fictive". In projects with a large dependency tree, things are usually more complicated (a more practical example could be cobra, which through 3 levels of indirection also depends on grpc, and etcd, and, well, "the world").

These issues often can arise from some indirect dependency deep in the dependency tree. More often than not, some small module that's (e.g.) following "best practice", and enables dependabot to always keep its dependencies on the most current version. That module itself may not be affected at all by new updates (it may only be using a small bit of it, or perhaps only as part of a tools.go (those are fun!)). But as a result, that tiny dependency forces the whole dependency chain to update to the latest version (effectively nullifying the "minimal version selection" of go modules).

Now, these things can be solved in "my project" (say, containerd); we can add an exclude rule (probably those will be "many" if one is needed for each version you don't want?) or use a replace rule (pin it to a specific version), but besides having to maintain the list of exclude (which can be potentially fun job), none of those help any of the consumers of my module. Effectively, it's kicking the can down the road, making it "whoever's next's problem".

In the ideal situation, things "explode", and their code fails to build, helping them notify "something's wrong!". Often, things are more subtle than that, and consumers may be living on a ticking timebomb (not theoretical; this happened on more than one occasion, sometimes even resulting in security issues).

If replace and/or exclude rules would be transitive, that would somewhat help (preventing to kick things down the road), but a "smarter" version resolution, even at the cost of a performance penalty (at time of version resolution) would be great.

I realise this is a tough problem to solve, and (to some extend), go modules may not have been designed for large projects, but it's the situation we're in, so trying to find what options are possible. As mentioned; some of these problems can be "solved" or hacked/worked around by large projects themselves, but these projects also have a responsibility towards their respective consumers (of which there may be many), so I'm also trying to explore if there's a solution that would address that side.

[liggitt]

version resolution only is needed when updating a dependency. It's a "one-time" performance penalty

I thought it ran on pretty much every go invocation (run, test, build, list, etc, etc)

[bcmills]

version resolution only is needed when updating a dependency. It's a "one-time" performance penalty

I thought it ran on pretty much every go invocation (run, test, build, list, etc, etc)

When lazy module loading is in effect (that is, if the main module is at go 1.17 or higher), we only load the full module graph when a package to be imported isn't found directly in the go.mod file (such as when running go test on some transitively imported package).

So, pretty much every go invocation can trigger a full load, but at 1.17 and above it's not as pervasive as it used to be.