Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go list fails with submodules which have test-only dependencies #60667

Closed
smira opened this issue Jun 7, 2023 · 17 comments
Closed

cmd/go: go list fails with submodules which have test-only dependencies #60667

smira opened this issue Jun 7, 2023 · 17 comments
Assignees
Labels
GoCommand cmd/go modules NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Milestone

Comments

@smira
Copy link

smira commented Jun 7, 2023

What version of Go are you using (go version)?

$ go version
go1.20.5

Does this issue reproduce with the latest release?

yes, I believe it is related to #60001

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/smira/.cache/go-build"
GOENV="/home/smira/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/smira/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/smira/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/smira/sdk/go1.20.5"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/smira/sdk/go1.20.5/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.5"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/smira/Documents/go-test-package-failure/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build337276505=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I hit this on a big repository, but I reduced this down to https://github.com/smira/go-test-package-failure.

There's a submodule which has an import of a package from _test.go file.

What did you expect to see?

Switching to older version of Go "fixes" it:

$ ~/go/bin/go1.20.3 list -u -m -json all
{
	"Path": "github.com/smira/go-test-package-failure",
	"Main": true,
	"Dir": "/home/smira/Documents/go-test-package-failure",
	"GoMod": "/home/smira/Documents/go-test-package-failure/go.mod",
	"GoVersion": "1.20"
}
{
	"Path": "github.com/santhosh-tekuri/jsonschema/v5",
	"Version": "v5.3.0",
	"Time": "2023-04-04T18:54:33Z",
	"Indirect": true,
	"Dir": "/home/smira/go/pkg/mod/github.com/santhosh-tekuri/jsonschema/v5@v5.3.0"
}
{
	"Path": "github.com/smira/go-test-failure/subpackage",
	"Version": "v0.0.0-00010101000000-000000000000",
	"Replace": {
		"Path": "./subpackage",
		"Dir": "/home/smira/Documents/go-test-package-failure/subpackage",
		"GoMod": "/home/smira/Documents/go-test-package-failure/subpackage/go.mod",
		"GoVersion": "1.20"
	},
	"Dir": "/home/smira/Documents/go-test-package-failure/subpackage",
	"GoMod": "/home/smira/Documents/go-test-package-failure/subpackage/go.mod",
	"GoVersion": "1.20"
}

What did you see instead?

In the checkout, the following fails:

$ ~/go/bin/go1.20.5 list -u -m -json all
go: updates to go.sum needed, disabled by -mod=readonly
@smira
Copy link
Author

smira commented Jun 7, 2023

Looking more into it, it might be related to github.com/santhosh-tekuri/jsonschema/v5 package, but I'm not sure what's exactly wrong with it

@dmitshur dmitshur changed the title go: go list fails with submodules which have test-only dependencies cmd/go: go list fails with submodules which have test-only dependencies Jun 7, 2023
@Nasfame
Copy link
Contributor

Nasfame commented Jun 7, 2023

Found a hot fix

  1. Remove the go version in go.mod
module github.com/smira/go-test-package-failure

replace github.com/smira/go-test-failure/subpackage => ./subpackage

require github.com/smira/go-test-failure/subpackage v0.0.0-00010101000000-000000000000

  1. bash go list -u -m -json all
image
  1. bash go mod download github.com/santhosh-tekuri/jsonschema/v5

  2. go list -u -m -json all

{
	"Path": "github.com/smira/go-test-package-failure",
	"Main": true,
	"Dir": "/workspaces/github-runner/test",
	"GoMod": "/workspaces/github-runner/test/go.mod",
	"GoVersion": "1.20"
}
{
	"Path": "github.com/santhosh-tekuri/jsonschema/v5",
	"Version": "v5.3.0",
	"Time": "2023-04-04T18:54:33Z",
	"Indirect": true,
	"Dir": "/go/pkg/mod/github.com/santhosh-tekuri/jsonschema/v5@v5.3.0",
	"GoMod": "/go/pkg/mod/cache/download/github.com/santhosh-tekuri/jsonschema/v5/@v/v5.3.0.mod",
	"GoVersion": "1.15"
}
{
	"Path": "github.com/smira/go-test-failure/subpackage",
	"Version": "v0.0.0-00010101000000-000000000000",
	"Replace": {
		"Path": "./subpackage",
		"Dir": "/workspaces/github-runner/test/subpackage",
		"GoMod": "/workspaces/github-runner/test/subpackage/go.mod",
		"GoVersion": "1.20"
	},
	"Update": {
		"Path": "github.com/smira/go-test-failure/subpackage",
		"Version": "v0.0.0-20230607191241-ca6da46c8544",
		"Time": "2023-06-07T19:12:41Z"
	},
	"Dir": "/workspaces/github-runner/test/subpackage",
	"GoMod": "/workspaces/github-runner/test/subpackage/go.mod",
	"GoVersion": "1.20"
}

@smira
Copy link
Author

smira commented Jun 8, 2023

Found a hot fix

I believe go mod tidy would remove this from go.mod, as it's unused.

@Nasfame
Copy link
Contributor

Nasfame commented Jun 8, 2023

@smira Tried it go mod tidy ; and yes it removes from go.mod. But the go list -u -m -json all still works

It still works with git reset --hard .

@smira
Copy link
Author

smira commented Jun 8, 2023

@Nasfame I can't reproduce it, go mod tidy removes the line from go.sum and things keep failing.

Basically the bug is that go mod tidy and go list are in disagreement on what should be in go.sum.

@dr2chase
Copy link
Contributor

dr2chase commented Jun 8, 2023

@bcmills @ianlancetaylor @rsc

@dr2chase dr2chase added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 8, 2023
@dmitshur dmitshur added the GoCommand cmd/go label Jun 8, 2023
@bcmills
Copy link
Member

bcmills commented Jun 8, 2023

Duplicate of #56222

@bcmills bcmills marked this as a duplicate of #56222 Jun 8, 2023
@bcmills bcmills closed this as not planned Won't fix, can't repro, duplicate, stale Jun 8, 2023
@smira
Copy link
Author

smira commented Jun 8, 2023

I'm not sure why it's a duplicate, the problem comes only with #56222 fix backported to Go 1.20 in Go 1.20.5. It worked fine before that.

@bcmills
Copy link
Member

bcmills commented Jun 8, 2023

Ah, I had missed that detail. Yes, it seems that the backported fix needs an extra condition for when to save the checksums.

@bcmills bcmills reopened this Jun 8, 2023
@bcmills bcmills self-assigned this Jun 8, 2023
@bcmills bcmills added modules NeedsFix The path to resolution is known, but the work has not been done. labels Jun 8, 2023
@bcmills bcmills added this to the Go1.21 milestone Jun 8, 2023
@gopherbot gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 8, 2023
@lespea
Copy link

lespea commented Jun 8, 2023

I see this is marked for 1.21, when this is resolved will this be backported to 1.20.6? This is breaking a lot of our tooling and we won't be able to update to 1.21+ for the foreseeable future giving that go is dropping support for a lot of operating system versions.

@bcmills
Copy link
Member

bcmills commented Jun 8, 2023

Yes, I intend for it to be. (That, or we will roll back part of all of #60001 and #60000.)

@gopherbot
Copy link

Change https://go.dev/cl/502015 mentions this issue: cmd/go: omit checksums for go.mod files needed for go version lines more often in pre-1.21 modules

@bcmills
Copy link
Member

bcmills commented Jun 8, 2023

@gopherbot, please backport to Go 1.19 and 1.20. This is a regression introduced in a previous patch release.

@gopherbot
Copy link

Backport issue(s) opened: #60697 (for 1.19), #60698 (for 1.20).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

@bcmills
Copy link
Member

bcmills commented Jun 8, 2023

In the meantime you can work around this with one of the following:

  • build from source and patch in the proposed fix CLs (once I've resolved the merge conflicts for the backports)
  • set -mod=mod in the go list command (to allow it to update the go.sum file)
  • run go mod tidy -compat=1.16 (instead of just go mod tidy) to record all of the needed checksums upfront.

@gopherbot
Copy link

Change https://go.dev/cl/502016 mentions this issue: [release-branch.go1.20] cmd/go: omit checksums for go.mod files needed for go version lines more often in pre-1.21 modules

@gopherbot
Copy link

Change https://go.dev/cl/502017 mentions this issue: [release-branch.go1.19] cmd/go: omit checksums for go.mod files needed for go version lines more often in pre-1.21 modules

joergjo added a commit to joergjo/go-samples that referenced this issue Jun 9, 2023
gopherbot pushed a commit that referenced this issue Jun 13, 2023
…d for go version lines more often in pre-1.21 modules

This updates the logic from CL 489075 to avoid trying to save extra
sums if they aren't already expected to be present
and cfg.BuildMod != "mod" (as in the case of "go list -m -u all" with
a go.mod file that specifies go < 1.21).

Fixes #60698.
Updates #60667.
Updates #56222.

Change-Id: Ied6ed3e80a62f9cd9a328b43a415a42d14481056
Reviewed-on: https://go-review.googlesource.com/c/go/+/502016
Reviewed-by: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
gopherbot pushed a commit that referenced this issue Jun 13, 2023
…d for go version lines more often in pre-1.21 modules

This updates the logic from CL 489075 to avoid trying to save extra
sums if they aren't already expected to be present
and cfg.BuildMod != "mod" (as in the case of "go list -m -u all" with
a go.mod file that specifies go < 1.21).

Fixes #60697.
Updates #60667.
Updates #56222.

Change-Id: Ied6ed3e80a62f9cd9a328b43a415a42d14481056
Reviewed-on: https://go-review.googlesource.com/c/go/+/502017
Reviewed-by: Michael Matloob <matloob@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Bypass: Bryan Mills <bcmills@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GoCommand cmd/go modules NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Projects
None yet
Development

No branches or pull requests

7 participants