cmd/go: mod_lazy_new_import and mod_multirepo flaky failed on linux-wsl and linux-s390x builders

[mengzhuo]

Does this issue reproduce with the latest release?

maybe, it failed at 77d9252

What did you expect to see?

All test passed

What did you see instead?

https://build.golang.org/log/8f84b20723e5bdae8362924b3a103ced90a515c6
https://build.golang.org/log/87edf847a20c338f407a5183c1fd6c80e070555e

go test proxy running at GOPROXY=http://127.0.0.1:45001/mod
--- FAIL: TestScript (0.01s)
    --- FAIL: TestScript/mod_lazy_new_import (0.11s)
        script_test.go:272: 
            # (2022-06-07T15:13:22Z)
            # This test illustrates the use of a deepening scan to resolve transitive
            # imports of imports of new packages from within existing dependencies.
            # The package import graph used in this test looks like:
            #
            # lazy ---- a/x ---- b
            #     \
            #      ---- a/y (new) ---- c
            #
            # Where a/x and a/y are disjoint packages, but both contained in module a.
            #
            # The module dependency graph initially looks like:
            #
            # lazy ---- a.1 ---- b.1
            #              \
            #               c.1 (0.019s)
            # Before adding a new import, the go.mod file should
            # enumerate modules for all packages already imported. (0.016s)
            # When we add a new import of a package in an existing dependency,
            # and that dependency is already tidy, its transitive dependencies
            # should already be present. (0.047s)
            # Now, we repeat the test with a lazy main module. (0.000s)
            # Before adding a new import, the go.mod file should
            # enumerate modules for all packages already imported. (0.008s)
            # When a new import is found, we should perform a deepening scan of the existing
            # dependencies and add a requirement on the version required by those
            # dependencies — not re-resolve 'latest'. (0.015s)
            > cp lazy.go.new lazy.go
            > ! go list all
            [stdout]
            example.com/a/x
            example.com/b
            example.com/lazy
            FAIL: testdata/script/mod_lazy_new_import.txt:52: unexpected command success
            

[cherrymui]

cc @bcmills

[cherrymui]

Also fails on the S390X builder
2022-06-07T17:36:56-81033fb/linux-s390x-ibm
2022-06-07T16:39:37-429a404/linux-s390x-ibm
2022-06-07T14:47:46-77d9252/linux-amd64-wsl
2022-06-06T15:42:19-47e34ca/linux-amd64-wsl
2022-06-04T18:42:07-8659114/linux-s390x-ibm
2022-06-03T20:15:20-162b882/linux-s390x-ibm

TestScript/mod_multirepo is also flaky on these two builders. Not sure if there is any connection.

go test proxy running at GOPROXY=http://127.0.0.1:38219/mod
--- FAIL: TestScript (0.01s)
    --- FAIL: TestScript/mod_multirepo (0.03s)
        script_test.go:272: 
            # (2022-06-07T17:04:43Z)
            # initial standalone module should use no downloaded modules (0.005s)
            # v2 import should use a downloaded module
            # both without an explicit go.mod entry ... (0.025s)
            > cp tmp/use_v2.go x.go
            > go get .
            > go list -deps -f {{.Dir}}
            [stdout]
            $WORK/gopath/src
            [stderr]
            x.go:2:8: no required module provides package rsc.io/quote/v2; to add it:
            	go get rsc.io/quote/v2
            [exit status 1]
            FAIL: testdata/script/mod_multirepo.txt:11: unexpected command failure
            
FAIL
FAIL	cmd/go	19.200s

2022-06-07T16:57:52-835a946/linux-s390x-ibm
2022-06-06T20:48:03-a71ca3d/linux-s390x-ibm
2022-06-06T19:53:39-c1e2ecb/linux-s390x-ibm
2022-06-06T19:03:15-95547ae/linux-s390x-ibm
2022-06-06T18:50:50-d43ddc1/linux-s390x-ibm
2022-06-06T18:24:19-07eca49/linux-s390x-ibm
2022-06-06T16:26:54-770146d/linux-s390x-ibm
2022-06-04T21:17:03-a32a592/linux-amd64-wsl
2022-06-03T20:16:45-54bd44e/linux-s390x-ibm

[jonathan-albrecht-ibm]

I'm able to reproduce this on a local s390x machine. I've been running one of the failing tests go test -count=100 cmd/go -run=Script/^mod_multirepo$ and it fails about 2 - 4 times in 100 runs.

@matloob, it started failing soon after https://go-review.googlesource.com/c/go/+/403778. I changed GOINDEX=true -> GOINDEX=false in src/cmd/go/script_test.go and then did not see any failures in over 1000 runs. So maybe related?

The failure in the mod_multirepo test seems to be because the go get . doesn't download anything. When that command is successful it has output like:

... SNIP ...
        > go get .
        [stderr]
        go: downloading rsc.io/quote/v2 v2.0.1
... SNIP ...

I tried adding -v -x options, ie. go get -v -x ., to see if there was any output at all when it failed but there wasn't.

Not sure what to try next so happy to try any suggestions. I can also provide access to an s390x VM if it would help.

[cherrymui]

@jonathan-albrecht-ibm thanks for looking into this!

Tentatively mark as a release blocker, as this is new failure. @matloob and @bcmills can decide. Thanks.

[matloob]

Hi, access to a s390x VM would be super helpful, thanks!

[jonathan-albrecht-ibm]

@matloob no problem. Please send me an email at the address on my profile page https://github.com/jonathan-albrecht-ibm and I'll send you the VM access details.

[jonathan-albrecht-ibm]

@matloob, I did a git bisect starting from before the first modindex changes were merged and got: ee87cd1 is the first bad commit.

I tested with the -testwork option and it when the test fails if I run go get . in the test tmp dir it keeps failing to get anything. But if I touch either of the x.go or go.mod src files, it will start working (ie. download the packages from the import in x.go).

I have tried debugging it a bit but haven't figured out exactly what's going on. I think its not building a new mod index in the failure case and some how using a default empty index. In the failure case, on modindex/read.go:133: data, _, err := cache.Default().GetMmap(id) doesn't return an error so data, err = indexModule(modroot) doesn't get called so there's nothing to do.

Hope that helps

[matloob]

It looks like the moduleHash isn't changing in TestScript/mod_multirepo after new contents of those files are copied in.

[gopherbot]

Change https://go.dev/cl/412394 mentions this issue: cmd/go/internal/modindex: disable indexing for modules outside GOROOT and the module cache

[bcmills]

I think this should be fixed by CL 412394, but I'd like to see more test runs on these builders to confirm.

[bcmills]

Does appear to be fixed by CL 412394, but that also had a typo that I missed and needs CL 412779 as a followup.

[jonathan-albrecht-ibm]

FWIW, I've been running the cmd/go tests on a local s390x machine at CL 412779 and no failures in ~500 runs. Before either CL 412394 or CL 412779 it would fail about once every five runs on this machine.

[bcmills]

CL 412779 is in, so we should be all clear on these failures now.