Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: do not allow the main module to replace (to or from) itself #34417

Open
rselph-tibco opened this issue Sep 19, 2019 · 10 comments

Comments

@rselph-tibco
Copy link

commented Sep 19, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13 darwin/amd64

This problem is also seen on Linux using go1.12.3

$ go version
go version go1.12.3 linux/amd64

And on Mac with the tip branch

$ gotip version
go version devel +fe2ed50 Thu Sep 19 16:26:58 2019 +0000 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/rselph/Library/Caches/go-build"
GOENV="/Users/rselph/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/rselph/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/local/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/local/lib/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="/usr/bin/clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/4w/6dmz_gmd6fj_qy_nwdwclg8c0000gp/T/go-build675848548=/tmp/go-build -gno-record-gcc-switches -fno-common"
GOROOT/bin/go version: go version go1.13 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.13
uname -v: Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.14.6
BuildVersion:	18G95
lldb --version: lldb-1001.0.13.3
  Swift-5.0

What did you do?

Option A:

$ git clone https://github.com/rselph-tibco/go-unstable-mods.git
$ cd go-unstable-mods
$ git checkout start_here
$ git switch -c new_branch
$ ./run.sh

The git repository will be updated with the results of each step. Optionally, comment out the git
commands to simply produce the error without recording results along the way.

This is equivalent to:

  1. Start with the contents of the attached file
    go-unstable-mods-start_here.tar.gz
  2. Set GOPATH and GOCACHE to point at empty directories
  3. From the sample1 directory run go mod tidy
  4. From the sample2 directory run go mod tidy
  5. From the sample2 directory run go install ./...
  6. From the sample1 directory run go install ./...
  7. Repeat the last step indefinitely

At this point, sample1/go.mod will never stabilize.

What did you expect to see?

go.mod should stabilize when the build is given the same inputs over and over.

What did you see instead?

go.mod eventually oscillates between two states, preventing -mod readonly from ever working, and wreaking
havoc with source control.

@jayconrod jayconrod changed the title go.mod changes on each build, -mod=readonly can never work cmd/go: go.mod changes on each build, -mod=readonly can never work Sep 19, 2019
@jayconrod jayconrod added this to the Go1.14 milestone Sep 19, 2019
@jayconrod

This comment has been minimized.

Copy link
Contributor

commented Sep 19, 2019

I took a quick look at this, need to investigate further though.

The first go install ./... command in sample1 removes a bunch of requirements from go.mod. The only reason that would happen is if the requirements are implied or redundant with some transitive requirement. Since each module is replacing itself and requiring the other module at an invalid version, I think sample1 is seeing its own go.mod file as a go.mod in a different version of sample1, so it sees its own requirements as redundant.

As I said, need to investigate further, but this seems like a strange edge case of replace. We should forbid modules from replacing themselves without specifying a version.

@rselph-tibco

This comment has been minimized.

Copy link
Author

commented Sep 19, 2019

Thanks for taking a look. If I understand you correctly, a workaround might be to specify the versions of sample1 and sample2 in the replace statements. That's certainly something I can try. I'm worried about the implications, though, since what I'm modeling is two modules under active development locally. Would I need to increment the versions each time the source changes to deal with cache issues?

@thepudds

This comment has been minimized.

Copy link

commented Sep 20, 2019

I’m not quite following this example, but why are the modules replacing themselves?

Regarding:

what I'm modeling is two modules under active development locally. Would I need to increment the versions each time the source changes to deal with cache issues?

Again, I am not quite following the example, but I think you shouldn’t need to increment versions if you are doing local filesystem-based replaces of modules that are being locally developed.

This FAQ has an example:

https://github.com/golang/go/wiki/Modules#can-i-work-entirely-outside-of-vcs-on-my-local-filesystem

Sorry for the quick comment, and sorry if it is not helpful.

@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 20, 2019

@rselph-tibco, it should be very rare for a module to need to replace itself, since the main module itself is always considered higher-precedence than any explicit version of its own path.

The only reason I can think of for a module to replace itself would be if some other interdependent module depends on an invalid version of it, in which case it should suffice to replace the invalid version with a valid one.

In the rare case in which you are developing two entirely new interdependent modules from scratch, #33370 may be relevant.

@bcmills bcmills changed the title cmd/go: go.mod changes on each build, -mod=readonly can never work cmd/go: do not allow the main module to replace its own path without an explicit version Sep 20, 2019
@rselph-tibco

This comment has been minimized.

Copy link
Author

commented Sep 20, 2019

Hi, thanks for all the comments. I arrived at the modules replacing themselves only because without it, I get this when building one module or the other (in this case, sample2):

go install ./...
++++++++++++++++++++++++++
go: foo.com/me/sample1@v0.0.0-00010101000000-000000000000 requires
	foo.com/me/sample2@v0.0.0-00010101000000-000000000000: unrecognized import path "foo.com/me/sample2" (https fetch: Get https://foo.com/me/sample2?go-get=1: dial tcp 23.23.86.44:443: connect: connection refused)
@rselph-tibco

This comment has been minimized.

Copy link
Author

commented Sep 20, 2019

BTW, I just verified that adding version v0.0.0-00010101000000-000000000000 explicitly to the replace directives didn't change the behavior with 1.13 or tip. But I agree @bcmills, that #33370 sounds related.

@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 20, 2019

BTW, I just verified that adding version v0.0.0-00010101000000-000000000000 explicitly to the replace directives didn't change the behavior with 1.13 or tip.

Yeah, that's not surprising: it makes go mod tidy think that the dependency on the zero-version module (via sample2) provides all of the same requirements as the main module, and thus the main module's requirements are irrelevant.

So it removes those requirements from the main module, but because of the replace directive that also removes the transitive requirements upon which the removal relied.

The core of the problem is that the module system assumes that versions are immutable, but replace directives make them mutable — and, in this case, go mod tidy actually performs the mutation itself.

Probably that implies that we should also not allow the target of the replace directive to be the directory containing the main module.

@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 20, 2019

The corrected version of this example, I think, is to replace v0.0.0-[…] with a directory containing only a go.mod file that specifies no dependencies:

replace foo.com/me/sample1 v0.0.0-00010101000000-000000000000 => ./empty-sample1
@bcmills bcmills changed the title cmd/go: do not allow the main module to replace its own path without an explicit version cmd/go: do not allow the main module to replace (to or from) itself Sep 20, 2019
@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 20, 2019

The two diagnostic fixes we can make for this are:

  • Emit an error if the main module contains a replace directive that matches its own path and version.
  • Emit an error if the main module contains a replace directive that resolves to the filesystem directory containing its own go.mod file.
@bcmills bcmills added the help wanted label Sep 20, 2019
@rselph-tibco

This comment has been minimized.

Copy link
Author

commented Sep 20, 2019

It looks like the empty directory trick works. I'll give this a try in our actual build environment. Thanks.

@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.