New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: incorrect sum for replaced module in either 'mod tidy' or 'mod vendor' #27868

Open
kevinburke opened this Issue Sep 25, 2018 · 10 comments

Comments

Projects
None yet
5 participants
@kevinburke
Contributor

kevinburke commented Sep 25, 2018

Please answer these questions before submitting your issue. Thanks!

What did you do?

Clone the repo at https://github.com/kevinburke/go-mod-nondeterministic.

Running GO111MODULE=on go mod tidy modifies go.sum.
Running GO111MODULE=on go mod vendor changes go.sum back to the original value.

What did you expect to see?

I expect go.sum to reach a steady state at some point, where further command invocations make no changes to the file.

What did you see instead?

go.sum changes after running each of go mod tidy and go mod vendor. There's no "steady state" since the file state depends on the last command that was run.

Does this issue reproduce with the latest release (go1.11)?

Yes

System details

go version devel +93ad702251 Tue Sep 25 22:07:55 2018 +0000 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/kevin/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/kevin"
GOPROXY=""
GORACE=""
GOROOT="/Users/kevin/go"
GOTMPDIR=""
GOTOOLDIR="/Users/kevin/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
GOROOT/bin/go version: go version devel +93ad702251 Tue Sep 25 22:07:55 2018 +0000 darwin/amd64
GOROOT/bin/go tool compile -V: compile version devel +93ad702251 Tue Sep 25 22:07:55 2018 +0000
uname -v: Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.13.6
BuildVersion:	17G65
lldb --version: lldb-1000.11.37.1
  Swift-4.2

@kevinburke kevinburke added the modules label Sep 25, 2018

@kevinburke

This comment has been minimized.

Contributor

kevinburke commented Sep 25, 2018

In addition, I get the following error when running go mod why:

$ GO111MODULE=on go mod why github.com/sourcegraph/zoekt
go: finding github.com/sourcegraph/zoekt latest
go: github.com/sourcegraph/zoekt@v0.0.0-20180925141536-852b3842c11d: parsing go.mod: unexpected module path "github.com/google/zoekt"
go: error loading module requirements

Please let me know if I should file a second issue for that.

@myitcv

This comment has been minimized.

Member

myitcv commented Sep 26, 2018

The delta appears within github.com/sourcegraph/zoekt so this is almost certainly some function of the replace directive that exists in the go.mod:

diff --git a/go.sum b/go.sum
index 1d2e68e..034fb23 100644
--- a/go.sum
+++ b/go.sum
@@ -26,7 +26,6 @@ github.com/pelletier/go-buffruneio v0.2.0/go.mod h1:JkE26KsDizTr40EUHkXVtNPvgGtb
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
-github.com/sourcegraph/zoekt v0.0.0-20180925141536-852b3842c11d h1:82xTRg5XBKlVsvLvgcgQzcG54AFTYhS2DdMVT5U5AWw=
 github.com/sourcegraph/zoekt v0.0.0-20180925141536-852b3842c11d/go.mod h1:f1Qnbu4glSRTb9A/H7qLv4AhudFf3eStkDbpyb21KYQ=
 github.com/src-d/gcfg v1.3.0/go.mod h1:p/UMsR43ujA89BJY9duynAwIpvqEujIH/jFlfL7jWoI=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=

cc @rsc @bcmills

@bcmills bcmills added this to the Go1.12 milestone Sep 26, 2018

@bcmills bcmills self-assigned this Sep 26, 2018

@kevinburke

This comment has been minimized.

Contributor

kevinburke commented Sep 28, 2018

FWIW, I'm manually editing go.sum multiple times per day on a (soon to be, large, probably popular) open source project to work around this issue.

During a user test today I also spoke with someone who said they were sticking with go1.10 because they had a bad experience with Go modules.

@odiferousmint

This comment has been minimized.

odiferousmint commented Oct 2, 2018

I never run go build in the project's directory that I am going to git push because it changes the go.sum and adds lots of duplicates of dependencies with a different checksum (IIRC), otherwise I would have to remove go.sum and do go mod verify every time I want to git push. It is an annoyance and I am not sure why it does that when the vendor directory is intact.

@dmitris

This comment has been minimized.

Contributor

dmitris commented Oct 8, 2018

I'm having an issue with go.sum being modified as a side-effect of running go install ./... (or go mod why - but not with go mod tidy) - not sure if the same issue, let me know if it should go in a separate one. Go version is go1.11.1.

I have a go.mod file with 15 replace lines in the form:
replace github.com/ardielle/ardielle-go => mirror.foobar.com/ardielle/ardielle-go.git v1.5.1
replace github.com/aws/aws-sdk-go => mirror.foobar.com/aws/aws-sdk-go.git v1.15.49 // indirect
github.com/coreos/go-oidc => mirror.foobar.com/coreos/go-oidc.git v2.0.0+incompatible
github.com/golang/protobuf => mirror.foobar.com/golang/protobuf.git v1.2.0
etc.
I run go mod tidy and happily checking go.mod and go.sum. Re-running go mod tidy of course does not product any changes, as expect. But! - running go install -v ./... has a side effect of modifying go.sum which grows from 96 to 110 lines with the following lines being added:
+mirror.foobar.com/ardielle/ardielle-go.git v1.5.1 h1:s1uPWrHpT/9tcsN8To9KFgq1iOFrRn/0Db5bn7rmOWQ=
+mirror.foobar.com/aws/aws-sdk-go.git v1.15.49 h1:2wqyo+X4SGkh67paS1vqTCmS2+n/UmSYEOfAyhcnZvY= etc.
the old lines like with /go.mod suffix next to the version - like mirror/foobar.com/aws/aws-sdk-go.git v1.15.49/go.mod h1:mFuSZ37Z9YOHbQEwBWztmVzqXrEkub65tZoCYDt7FT0= - are still preserved

@dmitris

This comment has been minimized.

Contributor

dmitris commented Oct 8, 2018

@thepudds asked on Slack to post the samples of go.sum - attaching 3 files: go.mod.txt, go.sum.txt, and go.sum.modified.txt
with modifications described above introduced by running go install ./... (or go mod why -m <pkg>). Removed internal dependencies (not involved in the diffs) and replaced the mirror domain name with the fake mirror.foobar.com value.

go.mod.txt
go.sum.txt
go.sum.modified.txt

@bcmills bcmills changed the title from cmd/go: "go mod tidy" / "go mod vendor" can't agree on correct go.sum to cmd/go: incorrect sum for replaced module in either 'mod tidy' or 'mod vendor' Oct 9, 2018

@myitcv

This comment has been minimized.

Member

myitcv commented Oct 19, 2018

go mod tidy appears to be at fault here (there may be other issues at play), but here's a reproduction:

$ export GOPATH=$(mktemp -d)
$ export PATH=$GOPATH/bin:$PATH
$ cd $(mktemp -d)
$ go mod init example.com/hello
go: creating new go.mod: module example.com/hello
$ cat <<EOD >main.go
package main
import _ "github.com/gopherjs/gopherjs/js"
import _ "rsc.io/quote"

func main() {}
EOD
$ go mod edit -require=github.com/gopherjs/gopherjs@latest -replace=github.com/gopherjs/gopherjs=github.com/myitcv/gopherjs@go1.11
$ go mod tidy
go: finding github.com/gopherjs/gopherjs latest
...

Look at the number of lines in go.sum for both rsc.io/quote (not replaced) and GopherJS (replaced):

$ cat go.sum | grep quote
rsc.io/quote v1.5.2 h1:w5fcysjrx7yqtD/aO+QwRjYZOKnaM9Uh2b40tElTs3Y=
rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPXsUe+TKr0=
$ cat go.sum | grep gopherjs
github.com/myitcv/gopherjs v0.0.0-20181007174709-d1c34fc540e3/go.mod h1:X83RgTaopDGeqahgXrQ120BVA5Ztn+oW4kxNueqOja4=

The single line for github.com/myitcv/gopherjs looks wrong.

"Fix" this using go list all:

$ go list all >/dev/null
go: finding github.com/myitcv/gopherjs v0.0.0-20181007174709-d1c34fc540e3

and check again:

$ cat go.sum | grep quote
rsc.io/quote v1.5.2 h1:w5fcysjrx7yqtD/aO+QwRjYZOKnaM9Uh2b40tElTs3Y=
rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPXsUe+TKr0=
$ cat go.sum | grep gopherjs
github.com/myitcv/gopherjs v0.0.0-20181007174709-d1c34fc540e3 h1:gHMnSFk1naYj5G8j9hhiYQ7kKWbsHjjGSPI8sM+Fqt4=
github.com/myitcv/gopherjs v0.0.0-20181007174709-d1c34fc540e3/go.mod h1:X83RgTaopDGeqahgXrQ120BVA5Ztn+oW4kxNueqOja4=

Run go mod tidy again to see if our "fix" persists:

$ go mod tidy
$ cat go.sum | grep quote
rsc.io/quote v1.5.2 h1:w5fcysjrx7yqtD/aO+QwRjYZOKnaM9Uh2b40tElTs3Y=
rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPXsUe+TKr0=
$ cat go.sum | grep gopherjs
github.com/myitcv/gopherjs v0.0.0-20181007174709-d1c34fc540e3/go.mod h1:X83RgTaopDGeqahgXrQ120BVA5Ztn+oW4kxNueqOja4=

It doesn't; hence we conclude go mod tidy is over aggressively pruning go.sum.

The "fix" here could involve the use of any go command that effectively invokes a build step, hence install, build, list etc all work.

@dmitris

This comment has been minimized.

Contributor

dmitris commented Nov 8, 2018

@bcmills - do you (or anyone from the Go team) agree that go mod tidy needs to be rectified? Is anyone working on this? What are the chances this can be fixed in 1.12?

@dmitris

This comment has been minimized.

Contributor

dmitris commented Nov 8, 2018

maybe as a workaround it would be possible for go mod tidy to "internally" run an equivalent of go list all >& /dev/null to "effectively invoke a build step" and "untidy" its over-aggressive purning?

@bcmills

This comment has been minimized.

Member

bcmills commented Nov 8, 2018

@dmitris, I'm planning to investigate this for 1.12 during the freeze. If the fix is not too invasive, it should make the 1.12 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment