Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go get tries to download non-existing commit #33076

Closed
ayzu opened this issue Jul 12, 2019 · 21 comments
Closed

cmd/go: go get tries to download non-existing commit #33076

ayzu opened this issue Jul 12, 2019 · 21 comments
Labels

Comments

@ayzu
Copy link

@ayzu ayzu commented Jul 12, 2019

Please answer these questions before submitting your issue. Thanks!

What did you do?

mkdir project
cd project
export GO111MODULE=on
go mod init project
go get github.com/micro/go-plugins@v1.0.0

What did you expect to see?

successful download of dependency

What did you see instead?

go: finding github.com/nats-io/nats.go v1.8.2-0.20190607221125-9f4d16fe7c2d
go: github.com/nats-io/nats.go@v1.8.2-0.20190607221125-9f4d16fe7c2d: unknown revision 9f4d16fe7c2d
go: error loading module requirements

Does this issue reproduce with the latest release (go1.12.7)?

yes

System details

go version go1.12.7 darwin/amd64

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/a8327/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/a8327/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.12.6/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.12.6/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/09/xght07y913s83cqxlwc_x3z017ny38/T/go-build408435640=/tmp/go-build -gno-record-gcc-switches -fno-common"

Note:
The case is that github.com/micro/go-plugins@v1.0.0 has a mod file https://github.com/micro/go-plugins/blob/v1.0.0/go.mod which requires github.com/nats-io/go-nats v1.7.0

Repository github.com/nats-io/go-nats is archived and probably redirects to https://github.com/nats-io/nats.go.

However, github.com/nats-io/nats.go does not have commit 9f4d16fe7c2d at all. Probably, it was here at some point but then was removed. So go get tries download non-existing commit from the master branch instead of latest tag 1.8.1 in github.com/nats-io/nats.go.

The same error will pop up, if we specify the dependency in go.mod and run go build.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jul 12, 2019

This issue does not reproduce for me with go1.13beta1 and the provided steps.

project$ go1.13beta1 mod init project
go: creating new go.mod: module project

project$ go1.13beta1 get github.com/micro/go-plugins@v1.0.0
go: finding github.com/micro/go-plugins v1.0.0
go: downloading github.com/micro/go-plugins v1.0.0
go: extracting github.com/micro/go-plugins v1.0.0

project$

Nor does it reproduce if the @v1.0.0 suffix is dropped, presumably because proxy.golang.org still has the missing commit in its cache:

project$ go1.13beta1 get github.com/micro/go-plugins
go: finding github.com/micro/go-plugins v1.1.1
[…]
go: finding github.com/micro/micro v1.7.1-0.20190627121529-410a2eba67f1

project$

The failure mode with go1.13beta1 and GOPROXY=direct is different, apparently due to a tag deleted from k8s.io/client-go. (The error message below is accurate: I checked https://github.com/kubernetes/client-go, and there really is no such tag at this point.)

project$ go1.13beta1 mod init project
go: creating new go.mod: module project

project$ export GOPROXY=direct

project$ go1.13beta1 get github.com/micro/go-plugins@v1.0.0
go: finding github.com/micro/go-plugins v1.0.0
go get: github.com/micro/go-plugins@v1.0.0 requires
        github.com/micro/go-config@v0.14.0 requires
        k8s.io/client-go@v2.0.0-alpha.0.0.20190126161006-6134db91200e+incompatible: invalid pseudo-version: preceding tag (v2.0.0-alpha.0) not found

project$

I can reproduce the failure mode you observed with go1.13beta1, GOPROXY=direct, and no @v1.0.0 suffix:

project$ go1.13beta1 get github.com/micro/go-plugins
go: finding github.com/micro/go-plugins v1.1.1
go get: github.com/micro/go-plugins@v1.1.1 requires
        github.com/nats-io/nats.go@v1.8.2-0.20190607221125-9f4d16fe7c2d: unknown revision 9f4d16fe7c2d

project$

That comes from an explicit requirement here, and that error message is also accurate: the repository at https://github.com/nats-io/nats.go contains no such commit.


Sorry, but this is not a bug in the go command. The proxy.golang.org mirror is continuing to cache this content for the moment (as designed), and also as designed, GOPROXY=direct is reporting the accurate state of the underlying repositories. With the caching proxy bypassed, there isn't much we can do if someone deletes an existing tag or force-pushes their repo to remove a commit or version that is already in use.

@bcmills bcmills closed this Jul 12, 2019
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jul 12, 2019

@bcmills bcmills changed the title go get tries to download non-existing commit cmd/go: go get tries to download non-existing commit Jul 12, 2019
@bcmills bcmills added the modules label Jul 12, 2019
@derekcollison

This comment has been minimized.

Copy link

@derekcollison derekcollison commented Jul 12, 2019

There is no 1.8.2 release, so this is from a commit on master most likely that may no longer exist.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jul 12, 2019

Yes: judging by the pseudo-version, it is specifically a commit on master at some point after the v1.8.1 tag.

@ayzu

This comment has been minimized.

Copy link
Author

@ayzu ayzu commented Jul 12, 2019

Yes, I guess that's exactly the case. However, my question is why it tries to find this non-existing commit, instead of using existing latest tag 1.8.1? Wouldn't be that expected behaviour?

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jul 12, 2019

@ayzu, builds using Go modules are expected to be 100% reproducible. If the configuration required for a reproducible build cannot be obtained, that results in an error rather than a silent change in behavior.

Note that you can explicitly change the behavior by adding a replace directive in your main module, such as

replace github.com/nats-io/nats.go v1.8.2-0.20190607221125-9f4d16fe7c2d => github.com/nats-io/nats.go v1.8.1
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jul 12, 2019

(And note that, at least for now, you can explicitly set GOPROXY=https://proxy.golang.org and to be able to build both versions of github.com/micro/go-plugins in their original configuration.)

@ayzu

This comment has been minimized.

Copy link
Author

@ayzu ayzu commented Jul 12, 2019

@derekcollison @bcmills

There is no 1.8.2 release, so this is from a commit on master most likely that may no longer exist.

Yes, this is the exact issue. It tries to find non-existing commit and fails. Is it expected behaviour?

The dependency in go.mod file is github.com/nats-io/go-nats v1.7.0 This repository redirects to github.com/nats-io/nats.go

From wiki:

In this section, "latest" is the latest version with a semver tag, or the latest known commit if there are no semver tags.

In my understanding go get was supposed to download the latest tag (1.8.1 > 1.7.0) or at least the exact 1.7.0 version. Instead, it tries to download non-existing commit (not tag).

Moreover, this commit v1.8.2-0.20190607221125-9f4d16fe7c2d is not even the latest one. It was removed at some part in the past, and there were some more commits after. According to https://github.com/nats-io/nats.go/commits/master, the latest commit on master was yesterday, 11 July.

Could you please elaborate why go get has chosen it?

@derekcollison

This comment has been minimized.

Copy link

@derekcollison derekcollison commented Jul 12, 2019

Is it listed in the go.sum?

@ayzu

This comment has been minimized.

Copy link
Author

@ayzu ayzu commented Jul 12, 2019

Is it listed in the go.sum?

You mean in go-plugins repository? https://github.com/micro/go-plugins/blob/6eef94c098315c5cfc03334fbcb5f8d43b1ca04a/go.sum#L523

Yes, it is listed:

github.com/nats-io/go-nats v1.7.0 h1:oQOfHcLr8hb43QG8yeVyY2jtarIaTjOv41CGdF3tTvQ=
github.com/nats-io/go-nats v1.7.0/go.mod h1:+t7RHT5ApZebkrQdnn6AhQJmhJJiKAvJUio1PiiCtj0=

@ayzu

This comment has been minimized.

Copy link
Author

@ayzu ayzu commented Jul 12, 2019

@bcmills @derekcollison

There is something more interesting. If I have a direct dependency github.com/micro/go-plugins@v1.0.0 which has go.mod which requries github.com/nats-io/go-nats v1.7.0, go build will fail with error github.com/nats-io/nats.go@v1.8.2-0.20190607221125-9f4d16fe7c2d: unknown revision 9f4d16fe7c2d. So instead of downloadinggithub.com/nats-io/go-nats v1.7.0 it will redirect to different repository github.com/nats-io/nats.go and will try to download non-existing (and not-latest commit). Note, that version of go-plugins is v1.0.0 and two different repos go-nats and nats.go.

But if try to download the exact dependency github.com/nats-io/go-nats v1.7.0 directly: go get github.com/nats-io/go-nats@v1.7.0. It will successfully download the required version without redirection to the other repository.

module local/dep

go 1.12

require (
	github.com/nats-io/go-nats v1.7.0 // indirect
	github.com/nats-io/nats.go v1.7.0 // indirect
	github.com/nats-io/nkeys v0.1.0 // indirect
	github.com/nats-io/nuid v1.0.1 // indirect
)

So, it looks like behaviour differs for those use cases.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jul 12, 2019

@ayzu, it seems pretty clear that the version in question is coming from the go.mod file in github.com/micro/go-plugins v1.1.1. If that's not the version of go-plugins you expected, it may be due to #29121, which is fixed in go1.13beta1.

@ayzu

This comment has been minimized.

Copy link
Author

@ayzu ayzu commented Jul 12, 2019

@ayzu, it seems pretty clear that the version in question is coming from the go.mod file in github.com/micro/go-plugins v1.1.1. If that's not the version of go-plugins you expected, it may be due to #29121, which is fixed in go1.13beta1.

Yes, it comes from v.1.1.1 But I specifically identify the version in the command: go get github.com/micro/go-plugins@v1.0.0. And this version (v1.0.0) has a go.mod file which requires github.com/nats-io/go-nats v1.7.0 and not pseudo-tag v1.8.2.

So why v1.0.0 version uses go.mod file from other verion (v1.1.1)?

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jul 12, 2019

See #29121.

@ayzu

This comment has been minimized.

Copy link
Author

@ayzu ayzu commented Jul 12, 2019

@bcmills

Could you please specify how the mentioned #29121 relates to my question?

As far as I understand, the mentioned problem was that go mod assigned incorrect pseudo-tag (starting with v2.0.0) to a commit.

The issue was solved, and the commit was assigned a correct pseudo-version k8s.io/client-go v0.0.0-20181126152608-d082d5923d3c (because the module identifies itself as module k8s.io/client-go, so for go.mod it is not version 2 regardless of any tag).

In this case, we have a pseudo-tag 1.8.2. This pseudo-tag is correct because the dependency identifies itself as github.com/nats-io/go-nats and the latest tag is 1.8.1.

However, my question was - why do go get uses this commit in the first place? (instead of a tag 1.7.0 which was required in a go.mod file, or latest tag 1.8.1 or even latest commit in the master branch)

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jul 12, 2019

@ayzu, #29121 causes go get github.com/micro/go-plugins@v1.0.0 to instead fetch github.com/micro/go-plugins@latest. The configuration of github.com/micro/go-plugins@latest explicitly refers to v1.8.2-0.20190607221125-9f4d16fe7c2d, for which the underlying commit was deleted (presumably by a force-push).

@ayzu

This comment has been minimized.

Copy link
Author

@ayzu ayzu commented Jul 12, 2019

@bcmills

Ok, thank you. I was under assumption that the mentioned issue has happened because kubernetes didn't followed semver.

So there is a bug in 1.12: one specifies a version "go get some/package@v1.0.8", but 'go get' downloads the latest version instead. As far understand, this bug occur only in some cases?

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jul 12, 2019

Yes. Specifically, it occurs when the argument to go get is a module that contains no .go source files at its root and the -m flag is not set.

(The -m flag is removed in go1.13beta1, but the bug is fixed in it too.)

@ayzu

This comment has been minimized.

Copy link
Author

@ayzu ayzu commented Jul 12, 2019

@bcmills

Ok, thank for your time and effort. Nice to hear that this bag was already fixed.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jul 12, 2019

for which the underlying commit was deleted (presumably by a force-push).

It turns out that that commit never should have been resolvable in the first place (see #31191): github.com/nats-io/nats.go@9f4d16fe7c2d was a PR, and was merged as commit 3d6c21bef4f7 instead.

That at least gives you a more precise target for a replace directive, though. 🙂

@ayzu

This comment has been minimized.

Copy link
Author

@ayzu ayzu commented Jul 13, 2019

Ok, now I have got the title of the issue: 'go get @' should not fall back to 'latest' if does not provide

I.e., tag version does not have go files in the root directory, and therefore does not provide a package.

sebbalex added a commit to italia/developers-italia-backend that referenced this issue Oct 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.