Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: foo@v2.0.0+incompatible currently allows Semantic Import Versioning to be optional #32695

Closed
thepudds opened this issue Jun 19, 2019 · 8 comments
Assignees
Milestone

Comments

@thepudds
Copy link

@thepudds thepudds commented Jun 19, 2019

This is a spinout from "Factor 2" from #31543 ("cmd/go: creating v2+ modules has lower success rate than it could").

Compared to Go 1.11 and 1.12, current tip / 1.13 might have increased the modes of interaction that work with v2+ modules that have not adopted Semantic Import Versioning.

What version of Go are you using (go version)?

go 1.12.6

Does this issue reproduce with the latest release?

Yes, including tip as of yesterday (devel +bc27b64d Tue Jun 18 03:41:59 2019 +0000)

What did you do?

Observe people accidentally creating and using modules that have v2+ semver tags but that have not adopted Semantic Import Versioning.

For example, if there is a module example.com/foo that:

  • has a go.mod file (that is, it has adopted modules)
  • has a v2.0.0 semver tag as its latest tag
  • and its module line reads module example.com/foo (without the in theory required /v2)

then this still works for a module-based consumer, even though the module is in a "bad" state:

go get example.com.com/foo@v2.0.0+incompatible

This means people can and do create usable v2+ modules that did not adopt Semantic Import Versioning, and consumers can and do consume those "bad" modules.

However, other things such as upgrades or go get example.com.com/foo@latest do not work as expected, which leads to confusion.

What did you expect to see?

Some type of warning when consuming a v2+ module that has not adopted Semantic Import Versioning (or perhaps an error in the future after the ecosystem has more time to adapt and more tools exist to help prevent or catch mistakes).

What did you see instead?

Here is a concrete example. This uses github.com/pierrec/lz4:

  • v2.2.3 has an incorrect go.mod file missing /v2 in the module statement.
  • v2.0.6 seems to be when the "bad" go.mod was first introduced.
  • v2.0.5 is the tag immediately before the "bad" go.mod was introduced.

First, here is Go 1.12 successfully building a consumer importing the "bad" v2.0.6, as well as this shows a subsequent upgrade not working in 1.12.

cd $(mktemp -d)
export GOPATH=$(mktemp -d)
go mod init tempmod
echo 'require github.com/pierrec/lz4 v2.0.6+incompatible' >> go.mod
cat <<EOF > consumer.go
package consumer
import _ "github.com/pierrec/lz4"
EOF

# build works
go build
go list -m github.com/pierrec/lz4
# outputs:
# github.com/pierrec/lz4 v2.0.6+incompatible

# attempting to upgrade via 'go get -u' does not upgrade lz4
go get -u
go list -m github.com/pierrec/lz4
# outputs:
# github.com/pierrec/lz4 v2.0.6+incompatible

# attempting to upgrade via '@latest' downgrades
go get github.com/pierrec/lz4@latest
go list -m github.com/pierrec/lz4
# outputs:
# github.com/pierrec/lz4 v2.0.5+incompatible

Second, tip is also able to build using the "bad" v2.0.6, but now tip allows upgrading.

cd $(mktemp -d)
export GOPATH=$(mktemp -d)
gotip mod init tempmod
echo 'require github.com/pierrec/lz4 v2.0.6+incompatible' >> go.mod
cat <<EOF > consumer.go
package consumer
import _ "github.com/pierrec/lz4"
EOF

# build works
gotip build
gotip list -m github.com/pierrec/lz4
# outputs:
# github.com/pierrec/lz4 v2.0.6+incompatible

# upgrade now actually does upgrade to a newer "bad" version
gotip get github.com/pierrec/lz4@latest
gotip list -m github.com/pierrec/lz4
# outputs:
# github.com/pierrec/lz4 v2.2.3+incompatible

head -1 $(gotip list -f '{{.Dir}}' github.com/pierrec/lz4)/go.mod
# outputs:
# module github.com/pierrec/lz4
@thepudds thepudds added the modules label Jun 19, 2019
@thepudds

This comment has been minimized.

Copy link
Author

@thepudds thepudds commented Jun 19, 2019

https://golang.org/cl/181881 might help with this.

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Jun 19, 2019

Hmm, this is unfortunate. In direct mode, it seems we correctly don't list these versions:

$ export GOPROXY=direct
$ go list -m -versions github.com/pierrec/lz4
github.com/pierrec/lz4 v1.0.1 v2.0.1+incompatible v2.0.2+incompatible v2.0.3+incompatible v2.0.4+incompatible v2.0.5+incompatible

However, the proxy knows about them.

$ export GOPROXY=https://proxy.golang.org,direct
$ go list -m -versions github.com/pierrec/lz4
github.com/pierrec/lz4 v1.0.1 v2.0.1+incompatible v2.0.2+incompatible v2.0.3+incompatible v2.0.4+incompatible v2.0.5+incompatible v2.0.6+incompatible v2.0.7+incompatible v2.1.1+incompatible v2.1.2+incompatible v2.2.3+incompatible

And in your example, the Go command validation doesn't catch the problem with these versions, even with GOPROXY=direct. So I think what happened is someone manually wrote in these versions to a go.mod. The proxy fetched them, they passed validation, and the proxy serves them as normal versions thereafter.

Patching in https://golang.org/cl/181881, it looks like a go.mod with a version like this will break the build in direct mode, since it's now validated.

We should figure out how many modules have invalid requirements like this.

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Jun 19, 2019

@jayconrod jayconrod added this to the Go1.13 milestone Jun 19, 2019
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jun 19, 2019

Ugh. Agreed, we need to fix this and we need to figure out how big the crater will be.

@zikaeroh

This comment has been minimized.

Copy link

@zikaeroh zikaeroh commented Jun 20, 2019

I think another example of this is github.com/go-redis/redis, where its v6.15.3+incompatible is the first with a go.mod. I actually used tip just the other day to do an upgrade from 6.15.2 that 1.12 was unable to perform, thinking it was a bug that had been fixed in #30634 or similar. I guess I'm guilty of doing something I shouldn't have, but I didn't perform it by hand either.

EDIT: On a second look, setting GOPROXY=https://proxy.golang.org and doing an update the usual way on 1.12 also upgrades me to 6.15.3+incompatible, while direct does not.

@bcmills bcmills self-assigned this Jun 20, 2019
@bcmills bcmills added the NeedsFix label Jun 20, 2019
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jun 20, 2019

On go1.12.6 I get the following results:

example.com$ go1.12.6 mod download -json github.com/go-redis/redis@v6.15.3
go: finding github.com/go-redis/redis v6.15.3
{
        "Path": "github.com/go-redis/redis",
        "Version": "v0.0.0-20190503082931-75795aa4236d",
        "Info": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/cache/download/github.com/go-redis/redis/@v/v0.0.0-20190503082931-75795aa4236d.info",
        "GoMod": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/cache/download/github.com/go-redis/redis/@v/v0.0.0-20190503082931-75795aa4236d.mod",
        "Zip": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/cache/download/github.com/go-redis/redis/@v/v0.0.0-20190503082931-75795aa4236d.zip",
        "Dir": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/github.com/go-redis/redis@v0.0.0-20190503082931-75795aa4236d",
        "Sum": "h1:x7dvutgTPxsHoeUboXlOp9fEUMdIRO4OJV5ly1LYE40=",
        "GoModSum": "h1:W2YCLaZryXHirdd9QqwkiVUxCQsrx8SbLq9Uqk7JS7A="
}

example.com$ go clean -modcache
example.com$ go1.12.6 mod download -json github.com/go-redis/redis@v6.15.3+incompatible
go: finding github.com/go-redis/redis v6.15.3+incompatible
{
        "Path": "github.com/go-redis/redis",
        "Version": "v6.15.3+incompatible",
        "Info": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/cache/download/github.com/go-redis/redis/@v/v6.15.3+incompatible.info",
        "GoMod": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/cache/download/github.com/go-redis/redis/@v/v6.15.3+incompatible.mod",
        "Zip": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/cache/download/github.com/go-redis/redis/@v/v6.15.3+incompatible.zip",
        "Dir": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/github.com/go-redis/redis@v6.15.3+incompatible",
        "Sum": "h1:NZ0O90AhLSvSrvLZ/S9h7D4kl1mW2PrKyxL7MyBKO2g=",
        "GoModSum": "h1:W2YCLaZryXHirdd9QqwkiVUxCQsrx8SbLq9Uqk7JS7A="
}

That first result is the intended behavior; the second is this bug.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jun 20, 2019

Change https://golang.org/cl/181881 mentions this issue: cmd/go: validate pseudo-versions against module paths and revision metadata

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jul 15, 2019

Two more examples of modules that now fail validation:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.