Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go mod vendor does not produce a consistent modules.txt #28102

Closed
jsternberg opened this issue Oct 9, 2018 · 10 comments

Comments

Projects
None yet
4 participants
@jsternberg
Copy link

commented Oct 9, 2018

What did you do?

Ran go mod vendor and committed the results to source control. Then ran it in continuous integration by copying the folder, running go mod vendor, and diffing the folders with diff -qr vendor.orig/ vendor/ to find differences.

What did you expect to see?

I expected to see no differences.

What did you see instead?

There were many differences. Running diff -qr <(sort vendor.orig/modules.txt) <(sort vendor/modules.txt) caused it to report no differences between the files, so it appears the ordering was different.

System details

go version go1.11.1 darwin/amd64
GOARCH="amd64"
GOBIN="/Users/jsternberg/go/pkg/bin/github.com/influxdata/idpe"
GOCACHE="/Users/jsternberg/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/jsternberg/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.1/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/jsternberg/go/src/github.com/influxdata/idpe/go.mod"
GOROOT/bin/go version: go version go1.11.1 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.11.1
uname -v: Darwin Kernel Version 18.0.0: Wed Aug 22 20:13:40 PDT 2018; root:xnu-4903.201.2~1/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.14
BuildVersion:	18A391
lldb --version: lldb-1000.11.37.1
  Swift-4.2
@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 9, 2018

Running diff -qr <(sort vendor.orig/modules.txt) <(sort vendor/modules.txt) caused it to report no differences between the files, so it appears the ordering was different.

Different in what way? (Which modules were transposed, and does one of them transitively depend on the other?)

@gopherbot

This comment has been minimized.

Copy link

commented Oct 9, 2018

Change https://golang.org/cl/140862 mentions this issue: cmd/go/internal/mvs: document that BuildList is sorted

@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 10, 2018

It appears that the BuildList itself is sorted. If there is any nondeterminism, it likely comes from concurrent import scanning, but I haven't been able to reproduce it in a unit-test so far.

@jsternberg

This comment has been minimized.

Copy link
Author

commented Oct 10, 2018

I'll work on a reproducer for this today. If it helps you, it seems to occur for me when I use go mod vendor on a Mac, then I run it again within the golang:1.11 image in Docker (so on Linux). The version of 1.11.0 or 1.11.1 don't seem to make a difference.

@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 10, 2018

Ah. If you repeatedly run go mod vendor on a single platform, do you get consistent results?

@jsternberg

This comment has been minimized.

Copy link
Author

commented Oct 10, 2018

Yes, I think I do.

I'll attempt to verify with an example on a public repo when I can today or at least get more information on the difference in the private repo.

@jsternberg

This comment has been minimized.

Copy link
Author

commented Oct 10, 2018

I tried to produce a test to reproduce this and I found a combination of user-failure/a potentially additional command as needed.

The reason why I got the failure is because as part of CI, we were doing this:

$ cp -a vendor/ vendor.orig/
$ go mod vendor

When I copy vendor to another directory, it becomes part of the current project in some way and it starts resolving all of the vendored dependencies as if they were part of the current project. So I think my initial issue is wrong as that's user failure.

But, the reason why we do that is because in our CI, the place where we run that doesn't have git. We commit our vendor directory and we want to ensure our vendor directory is correctly committed. Our CI doesn't have git at this point, so I copied the vendor directory to contain it. It doesn't appear like there's a way to validate the vendor directory similar to dep check.

Am I missing a command or should I create a separate issue for that?

@mark-rushakoff

This comment has been minimized.

Copy link
Contributor

commented Oct 10, 2018

From go help packages:

Directory and file names that begin with "." or "_" are ignored
by the go tool, as are directories named "testdata".

_vendor.orig or .vendor.orig should keep go from thinking the copies are part of the project.

@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

It doesn't appear like there's a way to validate the vendor directory similar to dep check.

I think that's #27348.

@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

Closing, on the theory that @mark-rushakoff's workaround should address your immediate needs and #27348 should provide a longer-term solution.

Please let me know if there are any other parts that we need to keep track of.

@bcmills bcmills closed this Oct 11, 2018

gopherbot pushed a commit that referenced this issue Oct 26, 2018

cmd/go/internal/mvs: document that BuildList is sorted
Updates #28102

Change-Id: Iee1ff64c7720108d6d26bfbff60ea51877093960
Reviewed-on: https://go-review.googlesource.com/c/140862
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.