Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go get -u can't upgrade packages with invalid or non-existent versions in go.mod #45527

Open
matthewmueller opened this issue Apr 12, 2021 · 6 comments

Comments

@matthewmueller
Copy link

@matthewmueller matthewmueller commented Apr 12, 2021

What version of Go are you using (go version)?

$ go version
go version go1.16.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/m/Library/Caches/go-build"
GOENV="/Users/m/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/m/dev/pkg/mod"
GONOPROXY="github.com/go-duo/duo"
GONOSUMDB="github.com/go-duo/duo"
GOOS="darwin"
GOPATH="/Users/m/dev"
GOPRIVATE="github.com/go-duo/duo"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.3"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/m/dev/src/github.com/go-duo/welcome/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/4f/tcxcr6_55v9bp38d8g4hjlf80000gn/T/go-build2786980060=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I'm changing the location of source code for one of my dependencies, so I change the module path in go.mod.

go.mod

module github.com/org/repo

go 1.16

- require github.com/org/dep v0.0.0-20210410204338-0d7a392d906a
+ require gitlab.com/org/dep v0.0.0-20210410204338-0d7a392d906a

Note that I'm just changing the name using a find and replace, not the version information.

I was expecting to be able to run:

go get -u gitlab.com/org/dep

And it would update to whatever gitlab.com/org/dep has as its latest. Unfortunately, you run into this:

go: gitlab.com/org/dep@v0.0.0-20210410204338-0d7a392d906a: parsing go.mod:
        module declares its path as: github.com/org/dep
                but was required as: gitlab.com/org/dep

As a fallback, I try pointing to the SHA:

go get -u gitlab.com/org/dep@1ccc8c5848cd752f9b00c35cdafa8923bd9dcd22

I still see the same error.

go: gitlab.com/org/dep@v0.0.0-20210410204338-0d7a392d906a: parsing go.mod:
        module declares its path as: github.com/org/dep
                but was required as: gitlab.com/org/dep

FYI, I'm not certain this error is repeatable, this might be specific to my mistake of not replacing the imports. Regardless of this particular error, you will always get an error for having an invalid version in there, you can try v0.0.0-foo and you'll be greeted with an error.

What did you expect to see?

I feel like it's failing too early. In my mind, upgrade should automatically upgrade to a non-faulty version.

If go.mod needs to be validated to resolving semver, then at least pointing directly to a SHA/branch/latest should work regardless of what you have as a version in go.mod.

What did you see instead?

I think it should perform the upgrade, then update go.mod. No error. The current workaround is to delete the dependency then run the command:

module github.com/org/repo

go 1.16

- require github.com/org/dep v0.0.0-20210410204338-0d7a392d906a
go get -u gitlab.com/org/dep

This all seems a bit unnecessary.

@matthewmueller matthewmueller changed the title go get -u can't upgrade packages with invalid versions go get -u can't upgrade packages with invalid or non-existent versions in go.mod Apr 12, 2021
@mknyszek mknyszek changed the title go get -u can't upgrade packages with invalid or non-existent versions in go.mod cmd/go: go get -u can't upgrade packages with invalid or non-existent versions in go.mod Apr 12, 2021
@mknyszek mknyszek added this to the Backlog milestone Apr 12, 2021
@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Apr 12, 2021

Loading

@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Apr 12, 2021

Loading

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Apr 12, 2021

Just to confirm my understanding, is your intent to permanently change the module path (what we internally call a hard fork) or to temporarily fetch the module code from another location (soft fork)?

Soft forks are commonly used to make small changes, sometimes intended to be upstreamed. A replace directive is the right tool for that:

replace github.com/org/dep v0.0.0-20210410204338-0d7a392d906a => gitlab.com/org/dep v0.0.0-20210410204338-0d7a392d906a

You can add or remove replace directives with go mod edit.

Hard forks are more complicated. I wouldn't expect go get to be able to deal with this situation, since it requires not only changing the module path in go.mod upstream, but it also requires changing imports in your packages to use the new module path.

Loading

@bcmills
Copy link
Member

@bcmills bcmills commented Apr 12, 2021

I think what is happening here is:

  1. The hand-edit adds a requirement on gitlab.com/org/dep v0.0.0-20210410204338-0d7a392d906a, which fails to load due to a mismatched module path. (At that commit, the go.mod file still declares the github.com/org/dep path.)
  2. go get reports the mismatched module path as an error, because it is trying to preserve transitive dependencies of the existing requirement graph. (It can successfully identify gitlab.com/org/dep@latest, but it can't figure out how to incorporate that into the requirement graph because the graph is broken.)

I'm making the module loader more tolerant to errors for #36460, and I wonder to what extent this problem will go away with lazy loading. If we know we're upgrading to gitlab.com/org/dep@latest, and we can tolerate a broken module graph up until we add that upgrade, and we're in a lazy module so we get to assume that the transitive dependencies of the old path don't matter, then we should be able to do the upgrade without bothering to report the error for the (no-longer-relevant) old version.

Loading

@bcmills
Copy link
Member

@bcmills bcmills commented Apr 12, 2021

TL;DR: I'm hopeful that #36460 will solve this, but I ought to revisit this scenario when that feature is complete to double-check.

Loading

@bcmills bcmills self-assigned this Apr 12, 2021
@matthewmueller
Copy link
Author

@matthewmueller matthewmueller commented Apr 25, 2021

Just to confirm my understanding, is your intent to permanently change the module path (what we internally call a hard fork) or to temporarily fetch the module code from another location (soft fork)?

Sorry for the late reply here. Yes, I'm permanently changing the module path, but the upstream module's imports have already been adjusted from import "github.com/org/dep" to import "gitlab.com/org/dep".

So both github.com/org/dep and gitlab.com/org/dep are valid go-gettable modules, the only difference is the version. The gitlab repository was created after the github one, so the version number will be different.

I don't know what the gitlab version number is yet, so I expected to be able to run go get -u gitlab.com/org/dep or go get -u gitlab.com/org/dep@sha and it would sort it out for me.

Instead, because I have the old github version number, it's failing. This is probably the correct behavior in most cases because the version isn't valid for the gitlab repo, but I understand the meaning of go get -u to upgrade the dependency, so in this specific case, I don't think it should matter if the existing version in go.mod is invalid or incorrect.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants