cmd/go: package is replaced but not required

[karalabe]

What version of Go are you using (go version)?

$ go version
go version go1.16 linux/amd64

Does this issue reproduce with the latest release?

Yes

What did you do?

We have a fairly large repo (go-ethereum), which is a standalone application, but it can also be consumed as a library. We also have a code generator in our command suite which takes some API definitions and generates Go wrappers for them. The generated code relies on our main repository as a library.

The above is nothing special, but we'd like to have a test suite that uses the generator to wrap some APIs an then run them to make sure the generated code is executable and does what we expect. Currently the test uses the generator to create the wrapper Go files and then attempts to build it. Since the build depends on the main repo, we need to tell go (or go mod) about it.

Previously we used go mod init ... and then go mod edit --replace https://github.com/ethereum/go-ethereum=path/to/checkout to have the tester use the current working code (needed both for local development as well as PRs; that they always test the checked out code, not a pinned upstream version). Up until Go 1.15 this worked fine. Starting from Go 1.16, this setup is rejected.

What did you expect to see?

Go mod in 1.16 refuses to accept a replacement directive if the package being replaced isn't already part of the mod file. I could add a call to go get github.com/ethereum/go-ethereum or go mod edit --require github.com/ethereum/go-ethereum@master to the build, but that would do a network request and download out repo of >500MB. It makes no sense to have a test download external code when it's not even needed at all (the needed code is the checked out working copy the test runs from).

I guess my question is how can I convince Go mod to add a replace directive to a local copy without having it download an upstream copy first, which just gets discarded anyway.

What did you see instead?

callbackparam.go:10:2: module github.com/ethereum/go-ethereum provides package github.com/ethereum/go-ethereum and is replaced but not required; to add it:
      	go get github.com/ethereum/go-ethereum
[...]

[mvdan]

Try go mod tidy? I think that should not do a network request, as it should use the replacement module on your filesystem.

[mvdan]

It's also worth considering:

1. Should go get be doing a module download when the module is replaced with a local filesystem path? Intuitively, I think not. @karalabe can you confirm whether you checked this?

2. If I'm wrong and go get is meant to do a module download, should the advice above still use go get instead of something that won't do a download, like go mod tidy?

[karalabe]

I think I misdiagnosed my problem. Still investigating it. I'm seeing strange build times and I assumed it was downloading data, but maybe it's just doing some re-builds I'm not expecting.

[jayconrod]

go mod tidy and go get may both hit the network to look up imported packages that aren't provided by any required module. If a module is replace locally, the go command will look there first, but I think it may still go out to the network for other prefixes of the module path.

Instead, you can add a requirement on a non-existent version while replacing that version:

go mod edit -require example.com/mod@v0.0.0-local -replace example.com/mod@v0.0.0-local=../local

Adding a replacement, even one without a version on the left side, doesn't automatically add that module to the build list. If it did, the go command would read its go.mod file and apply its requirements. That could influence selected versions of other modules, even if the replaced module didn't provide any packages.

[bcmills]

go mod tidy should never do a network lookup if it could add a replaced module instead.¹

go get, on the other hand, will perform a network lookup in order to identify the true latest version, taking your replacements into account,² and then that version will be replaced instead of downloaded. It does that so that the latest version added by go get is always consistent with go list -m [⋯]@latest, and so that (if possible) your require directive always specifies a valid version for downstream consumers (if any), so that they won't break when they require your module. (Downstream consumers will not pick up your replace directives, so they need a valid version.)

If you are not using a proxy for the repo in question, that lookup may involve cloning the upstream repo. So that can be a pretty expensive operation. (Note that the official distributions of the go command use proxy.golang.org by default, but the Fedora fork of the go command does not.)

If that network lookup fails, then go get will also fall back to replacement versions.³

¹

go/src/cmd/go/internal/modload/import.go

Lines 334 to 351 in 7eb31d9

	for mp, mv := range index.highestReplaced {
	if !maybeInModule(path, mp) {
	continue
	}
	if mv == "" {
	// The only replacement is a wildcard that doesn't specify a version, so
	// synthesize a pseudo-version with an appropriate major version and a
	// timestamp below any real timestamp. That way, if the main module is
	// used from within some other module, the user will be able to upgrade
	// the requirement to any real version they choose.
	if _, pathMajor, ok := module.SplitPathVersion(mp); ok && len(pathMajor) > 0 {
	mv = modfetch.ZeroPseudoVersion(pathMajor[1:])
	} else {
	mv = modfetch.ZeroPseudoVersion("v0")
	}
	}
	mods = append(mods, module.Version{Path: mp, Version: mv})
	}

²

go/src/cmd/go/internal/modload/query.go

Lines 987 to 1000 in 7eb31d9

	repoVersions, err := rr.repo.Versions(prefix)
	if err != nil && !errors.Is(err, os.ErrNotExist) {
	return nil, err
	}
	versions := repoVersions
	if index != nil && len(index.replace) > 0 {
	path := rr.ModulePath()
	for m, _ := range index.replace {
	if m.Path == path && strings.HasPrefix(m.Version, prefix) && m.Version != "" && !modfetch.IsPseudoVersion(m.Version) {
	versions = append(versions, m.Version)
	}
	}
	}

³

go/src/cmd/go/internal/modload/query.go

Lines 1045 to 1062 in 7eb31d9

	if v, ok := index.highestReplaced[path]; ok {
	if v == "" {
	// The only replacement is a wildcard that doesn't specify a version, so
	// synthesize a pseudo-version with an appropriate major version and a
	// timestamp below any real timestamp. That way, if the main module is
	// used from within some other module, the user will be able to upgrade
	// the requirement to any real version they choose.
	if _, pathMajor, ok := module.SplitPathVersion(path); ok && len(pathMajor) > 0 {
	v = modfetch.PseudoVersion(pathMajor[1:], "", time.Time{}, "000000000000")
	} else {
	v = modfetch.PseudoVersion("v0", "", time.Time{}, "000000000000")
	}
	}
	if err != nil || semver.Compare(v, info.Version) > 0 {
	return rr.replacementStat(v)
	}
	}