cmd/go: upgrading a module gives different results based on "how" you upgrade

[marwan-at-work]

Summary

From the Go Wiki:

Day-to-day upgrading and downgrading of dependencies should be done using 'go get', which will automatically update the go.mod file. Alternatively, you can edit go.mod directly.

From go help modules

If go.mod is edited directly, commands
like 'go build' or 'go list' will assume that an upgrade is intended and
automatically make any implied upgrades and update go.mod to reflect them.

Therefore, people can upgrade a module by:

1. Simply running go get <mod>@<newVersion> or
2. Directly editing the go.mod file and replace the old version with the new one.

Most Go developers I've talked to, including myself, have believed that both options are interchangeable. However, whether a user picks option 1 or option 2 above can actually lead to a different final build list.

While I'm not sure this is a bug, but the fact that how you upgrade a module can lead to different results seems odd or at least something worth documenting.

Reproduce:

I have replicated the same dependency tree in the MVS article here: https://research.swtch.com/vgo-mvs

You can find that dependency tree under https://github.com/gomvs

github.com/gomvs/a is the main module and its current commit resides at Algorithm 1:

module github.com/gomvs/a

go 1.13

require (
	github.com/gomvs/b v1.2.0
	github.com/gomvs/c v1.2.0
)

If we want to trigger Algorithm 3 by upgrading one module (c v1.2.0 to c v1.3.0) we have 2 ways of doing it:

Option 1: go get github.com/gomvs/c@v1.3.0

This will properly implement Algorithm R by adding Module D v1.4.0 to the go.mod file because the Rough List still included c@v1.2.0

Therefore, the upgraded go.mod file will look like this:

module github.com/gomvs/a

go 1.13

require (
	github.com/gomvs/b v1.2.0
	github.com/gomvs/c v1.3.0
	github.com/gomvs/d v1.4.0 // indirect
)

If you run go run . you'll see the following dependency tree being called:

A
B v1.2.0
D v1.4.0
E v1.2.0
C v1.3.0
F v1.1.0
G v1.1.0

Option 2: Directly edit the go.mod file by going to line 7 in go.mod and simply changing v1.2.0 to v1.3.0

If a user decided to upgrade from c v1.2.0 to c v1.3.0 by "editing the go.mod file directly" then Algorithm R has no way of remembering that we had c v1.2.0 when running "go build" and therefore we end up downgrading to Module D v1.3.0.

The resulting go.mod file remains the same:

module github.com/gomvs/a

go 1.13

require (
	github.com/gomvs/b v1.2.0
	github.com/gomvs/c v1.3.0
)

And if you run go run . you get the following results:

A
B v1.2.0
D v1.3.0
E v1.2.0
C v1.3.0
F v1.1.0
G v1.1.0

Notice D v1.3.0 (and not D v1.4.0)

The MVS article mentions this behavior as "incorrect" and therefore should we consider directly upgrading the go.mod file incorrect? Also, should we document this behavior?

There is actually option 3, which I didn't know we could do:

Option 3: duplicate modules in go.mod file

You can actually have a go.mod file that looks like this:

module github.com/gomvs/a

go 1.13

require (
	github.com/gomvs/b v1.2.0
	github.com/gomvs/c v1.2.0
	github.com/gomvs/c v1.3.0
)

Notice, that c is mentioned twice at two different versions. Running go mod tidy ends up picking the correct version, and also adding d at v1.4.0 correctly. Given Algorithm R, this makes sense and I wonder if we should also mention it?

cc: @bcmills

[bcmills]

CC @jayconrod @thepudds

The MVS article mentions this behavior as "incorrect" and therefore should we consider directly upgrading the go.mod file incorrect?

Sort of? It's not “incorrect”, but it is a semantically different operation: editing the go.mod file essentially tells the go command, “use this exact set as my new dependencies”, whereas go get tells it “modify my dependencies so that these versions are used”.

The former resolves inconsistencies by taking the higher version, which potentially discards the edited versions, whereas the latter resolves inconsistencies by downgrading until the requested versions are actually what is selected.

I'm not sure that we should recommend adding duplicate lines as a standard practice. I could imagine a change to the go command that causes it to error out in case of inconsistencies rather than simply discarding inconsistent lines (see also #28692). Such a change would be backwards-compatible in some sense, since existing go commands do not write inconsistent go.mod files, but would break workflows that intentionally introduce inconsistencies in the go.mod file.

In my opinion, the most robust option is to always use go get.

[marwan-at-work]

editing the go.mod file essentially tells the go command, “use this exact set as my new dependencies”, whereas go get tells it “modify my dependencies so that these versions are used”.

I don't think that's mentioned anywhere in the wiki or docs, so should it be? As of recently, I thought the two operations would yield the same exact build list.

In fact the two quotes from the docs/wiki above suggest the two operations are interchangeable.

In my opinion, the most robust option is to always use go get.

👍 👍 👍

[matloob]

I think the documentation now fairly clearly reflects what go get does. I'm going to close this issue. Please reopen if you don't feel it's addressed.