cmd/go: allow replacement modules to alias other active modules

[mwf]

What version of Go are you using (go version)?

Go tip:
go version devel +f2131f6e0c Wed Aug 8 21:37:36 2018 +0000 darwin/amd64

Does this issue reproduce with the latest release?

Yes (it is not reproduced with go version go1.11beta2 darwin/amd64)

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/ikorolev/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/var/folders/_b/d1934m9s587_8t_6ngv3hnc00000gp/T/tmp.cqU8g8OM/gopath"
GOPROXY=""
GORACE=""
GOROOT="/Users/ikorolev/.gvm/gos/go1.11beta3"
GOTMPDIR=""
GOTOOLDIR="/Users/ikorolev/.gvm/gos/go1.11beta3/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/var/folders/_b/d1934m9s587_8t_6ngv3hnc00000gp/T/tmp.cqU8g8OM/vgo-a-user/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/_b/d1934m9s587_8t_6ngv3hnc00000gp/T/go-build138999780=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Sorry, no standalone reproduction, since issue is connected with repository forking

Assume we have a repository A: https://github.com/mwf/vgo-a with the only feature:

package a

var A = "A"

Than we have a fork1 https://github.com/mwf/vgo-a-fork1, adding a feature B :

package a

var B = "B is a new feature in a-fork1"

Unfortunately fork1 will never be merged to the upstream, just because a author don't like this feature.

It's important to note, that both a and a-fork1 don't have go.mod, they are too conservative for that 😄

Then we got a happy user, using both projects in his repo.
go.mod:

module github.com/mwf/vgo-a-user

require (
	github.com/mwf/vgo-a v0.1.0
	github.com/mwf/vgo-a-fork1 v0.2.0
)

main.go

package main

import (
	"fmt"

	"github.com/mwf/vgo-a"
	a_fork "github.com/mwf/vgo-a-fork1"
)

func main() {
	fmt.Printf("A: %q\n", a.A)
	fmt.Printf("B: %q\n", a_fork.B)
}

All just works fine:

$ go run .
A: "A"
B: "B is a new feature in a-fork1"

Here appears fork2 https://github.com/mwf/vgo-a-fork2, forked from fork1, and fixing some bugs both in the upstream and in fork1.

We use the fork2 with replace in our main repo: https://github.com/mwf/vgo-a-user/blob/master/go.mod

module github.com/mwf/vgo-a-user

require (
	github.com/mwf/vgo-a v0.1.0
	github.com/mwf/vgo-a-fork1 v0.2.0
)

replace github.com/mwf/vgo-a => github.com/mwf/vgo-a-fork2 v0.2.1

replace github.com/mwf/vgo-a-fork1 => github.com/mwf/vgo-a-fork2 v0.2.1

What did you expect to see?

Building this with go1.11beta2 works just fine:

cd `mktemp -d`
git clone git@github.com:mwf/vgo-a-user.git .
go version && go run .

Output:

go version go1.11beta2 darwin/amd64
go: finding github.com/mwf/vgo-a-fork2 v0.2.1
go: downloading github.com/mwf/vgo-a-fork2 v0.2.1
go: finding github.com/mwf/vgo-a v0.1.0
go: finding github.com/mwf/vgo-a-fork1 v0.2.0
A: "A, fixed in a-fork2"
B: "B, fixed in a-fork2"

What did you see instead?

Building with the tip (and beta3) returns an error:

cd `mktemp -d`
git clone git@github.com:mwf/vgo-a-user.git .
go version && go run .

Output:

go version devel +f2131f6e0c Wed Aug 8 21:37:36 2018 +0000 darwin/amd64
go: finding github.com/mwf/vgo-a-fork2 v0.2.1
go: downloading github.com/mwf/vgo-a-fork2 v0.2.1
go: github.com/mwf/vgo-a-fork1@v0.2.0 used for two different module paths (github.com/mwf/vgo-a and github.com/mwf/vgo-a-fork1)

More comments

I understand that this case is very specific and arguable - this should not ever happen ideally, but we have the real case here:
https://github.com/utrack/clay/blob/master/integration/binding_with_body_and_response/go.mod

There is a little workaround, to define go.mod at fork2 and make a replace upstream -> fork2_with_go.mod, but it's too dirty :)

replace github.com/mwf/vgo-a => github.com/mwf/vgo-a-fork2 v0.3.0 // version with go.mod
replace github.com/mwf/vgo-a-fork1 => github.com/mwf/vgo-a-fork2 v0.2.1 // no go.mod

It works with tip and beta3:

$ go version && go run .
go version devel +f2131f6e0c Wed Aug 8 21:37:36 2018 +0000 darwin/amd64
A: "A, fixed in a-fork2"
B: "B, fixed in a-fork2"

If you decide that the case is too specific and crazy, and you'd like to close as "Won't fix" - then I assume we should change the error string, because it's confusing now:

go: github.com/mwf/vgo-a-fork1@v0.2.0 used for two different module paths (github.com/mwf/vgo-a and github.com/mwf/vgo-a-fork1)

It should look like this:

go: github.com/mwf/vgo-a-fork2@v0.2.1 used for two different module paths (github.com/mwf/vgo-a and github.com/mwf/vgo-a-fork1)

because it's github.com/mwf/vgo-a-fork2 who's to blame for the error.

[mwf]

And this crazy stuff could happen only in non-go.mod repositories, because you can't use both upstream and fork at the same time if they have go.mod inside:

module github.com/mwf/vgo-a-user

require (
    github.com/mwf/vgo-a-mod v0.1.0
    github.com/mwf/vgo-a-mod-fork1 v0.2.0
)

$ go run .
go: finding github.com/mwf/vgo-a-mod-fork1 v0.2.0
go: github.com/mwf/vgo-a-mod-fork1@v0.2.0: parsing go.mod: unexpected module path "github.com/mwf/vgo-a-mod"
go: finding github.com/mwf/vgo-a-mod v0.1.0
go: error loading module requirements

You are forced either to use your fork as replace argument, or to change the module in fork's go.mod and never assume it as a fork again :) And this is for the best 👍

So no one will ever hit such an issue in the brave new world of go modules 😄

[bcmills]

Ideally, I think the long-term solution will be to treat replacements as rewriting the import paths rather than the source code, to allow for precisely this kind of fork-unification behavior.

(For an example of how this can go wrong otherwise, see the code in #26607.)

[mwf]

@bcmills you set "Go1.12" milestone for the issue.

Maybe we could at least fix the error string in Go1.11 to eliminate the confusion?

If you decide that the case is too specific and crazy, and you'd like to close as "Won't fix" - then I assume we should change the error string, because it's confusing now:

go: github.com/mwf/vgo-a-fork1@v0.2.0 used for two different module paths (github.com/mwf/vgo-a and github.com/mwf/vgo-a-fork1)

It should look like this:

go: github.com/mwf/vgo-a-fork2@v0.2.1 used for two different module paths (github.com/mwf/vgo-a and github.com/mwf/vgo-a-fork1)

because it's github.com/mwf/vgo-a-fork2 who's to blame for the error.

[bcmills]

Oh, yeah, that's an easy fix.

[gopherbot]

Change https://golang.org/cl/128878 mentions this issue: cmd/go/internal/modload: emit correct module in duplication error

[F21]

I am also seeing this problem when using go mod why. Since testcontainers/testcontainer-go was renamed to testcontainers/testcontainers-go recently, modules break for me since it is being used by one of my dependencies (not sure how deep).

I added:

replace github.com/testcontainers/testcontainer-go => github.com/testcontainers/testcontainers-go v0.0.0-20190108154635-47c0da630f72

to go.mod in order to get go get -u and go mod tidy to work correctly.

However, if I run go mod why github.com/testcontainers/testcontainers-go, I get:

go: finding github.com/testcontainers/testcontainers-go latest
go: github.com/testcontainers/testcontainers-go@v0.0.0-20190108154635-47c0da630f72 used for two different module paths (github.com/testcontainers/testcontainer-go and github.com/testcontainers/testcontainers-go)

[kevinburkemeter]

Just a heads up that one of the linked answers is the top result on Google when you search for "used for two different module paths" so it might be good to provide some easy guidance here on how to solve that problem. As it stands you have to read a lot to get your head around how this solves the issue.

[dmitshur]

This issue is currently labeled as early-in-cycle for Go 1.16.
That time is now, so this is a friendly ping so the issue is looked at again.

[bcmills]

I'm dropping the early-in-cycle, label on the theory that (empirically) hardly anybody tests the more advanced details of module mode until at least the beta anyway.

[mvdan]

To add another real case where this is causing pain: github.com/gopherjs/vecty has been moved to github.com/hexops/vecty, but dozens of library modules like github.com/marwan-at-work/vecty-router still require the old module. I tried upgrading vecty to master and then using replace github.com/gopherjs/vecty => github.com/hexops/vecty master to force third party libraries to come along for the upgrade, but encountered this issue.

cc @marwan-at-work in case he's open to fixing this in his module directly.

[marwan-at-work]

@mvdan done on the vecty-router side thanks to @NateWilliams2 marwan-at-work/vecty-router#12

[iamkirkbater]

Another case where this is causing pain:

An upstream dependency (A) is pointing to a git repo that has been deleted (B). Another dependency (C) is using a fork (D) of the repo that has been deleted. The fork (D) and the repo that has been deleted (B) are both exactly the same, just B no longer exists.

So a simplified Dep tree would look like this:

Our Project
  | - Dependency
  |   | - Dependency A
  |   |   | - Package B (now deleted)
  | - Dependency C
  |   | - Package D (Fork of B with no changes)

As a stopgap to continue our builds until the upstream dependency (A), we thought we could just replace B with D, but that's throwing the "X used for two different module paths", even though that's exactly what we want to do.

[Dragomir-Ivanov]

I have another issue with 1.15.3. Maybe I need to file another Issue, but will post here for now:
Having these in go.mod

replace gopkg.in/mgo.v2 => ../mgo

replace github.com/globalsign/mgo => ../mgo

Results in: go: ../mgo@ used for two different module paths (github.com/globalsign/mgo and gopkg.in/mgo.v2).
I feel that this should be allowed.

[bcmills]

@Dragomir-Ivanov, no need to file a separate issue. This is next on my list after #36460, although it might not make 1.16 before the freeze.

However, note that github.com/globalsign/mgo imports packages from itself. The replace directives that you describe, if they were allowed today, would result in two distinct copies of each package (one beginning with github.com/globalsign/mgo, and one beginning with gopkg.in/mgo.v2), but with each copy importing only one other copy (presumably github.com/globalsign/mgo?).

That could well produce type errors if types from the imported packages are used in the exported API, since each distinct package also defines its own types.

It seems that what you ideally need is exactly what we haven't implemented yet, which is a way to reinterpret import statements that refer to one of those copies as if they instead referred to the other.

[Dragomir-Ivanov]

@bcmills Thanks for your reply and all the efforts! I workaround my issues so far, by making private fork, and replaced all self import paths in github.com/globalsign/mgo to gopkg.in/mgo.v2.
In any case, I really think that once you replace a package, go toolchain should implicitly redirect replaced package self-imports to the replacer.

There are lots of abandoned packages, then revived by someone else and continued (as this mgo one), and we need a clean way to replace the package in the whole project(thus replacing package in our dependencies as well) without any forks, source modifications, etc.

I am in no rush what so ever, and can wait for 1.16, 1.17, etc.
Also I am thinking in what takes if we want to replace a package only for certain dependency packages. This can quickly become NPM :)

[fwhezfwhez]

go 1.12.17

used for two different module paths for github.com/golang/protobuf and golang.org/x/protobuf.

To fix this, I replace all my import path of 'golang.org/x/protobuf' to 'github.com/golang/protobuf'.

Sincerely I think these two import pkgs should support work together and not just force user to abandon one of it, because everything work fine in GOPATH and just go bad in GOMODULE. It's just not that smooth.

[Rikanishu]

Can anyone explain me why it's not possible to make go point to the same version of package from multiple aliases?
I don't understand the reason why a compilation process exits with the error. Looks like it's an inner go mod problem, not a something that should cause fail by design.

Will it be resolved somehow? Any progress of that?

Thanks!

[bcmills]

@Rikanishu, #26607 is why aliasing is currently disallowed. I plan to address this in Go 1.17.

[fwhezfwhez]

Can anyone explain me why it's not possible to make go point to the same version of package from multiple aliases?
I don't understand the reason why a compilation process exits with the error. Looks like it's an inner go mod problem, not a something that should cause fail by design.

Will it be resolved somehow? Any progress of that?

Thanks!

Maybe go chain will only find pkg source code by one and the only one specific key, this key is module-name.

Why two pkg sharing a same module-name will conflict. It(go chain) might use a map to storage a source code ref like:

refs map[string]*moduleRef

key is module-name, value is where source code put.

Here it meet a question!

• When analyse to import golang.org/x/protobuf, although it imported-path is named like this but its module name is github.com/golang/protobuf.Thus, it will be temporary stored as

if _, exist := refs["github.com/golang/protobuf"]; !exist  {
  refs["github.com/golang/protobuf"] = newModuleRef("GOPATH/pkg/mod/golang.org/x/protobuf")   // ✔
} else {
    panic("used for two different module path")
}

• When analyise to import github.com/golang/protobuf, its module name is also github.com/golang/protobuf. Then do a same operation

if _, exist := refs["github.com/golang/protobuf"]; !exist  {
  refs["github.com/golang/protobuf"] = newModuleRef("GOPATH/pkg/mod/github.com/golang/protobuf")   
} else {
    panic("used for two different module path")     // ✖
}

So only extend refs key format, this question can be solve then.