cmd/go: indexed packages not invalidated when location of parent go.mod file changes

[findleyr]

We have two concerning test failures on the builders, which seem to have the same root cause:

darwin-amd64-10_14: https://build.golang.org/log/88d3fe6fbcfe0ba7110aac537e5cdcae51737d4d
openbsd-386-70: https://build.golang.org/log/79c6a4815792817f70a447fecb4557aa33b90ff0

Both of these are concerning because gopls is failing to find package metadata for a file, even after reloading (and apparently successfully loading file data).

It's suprising that files are left orphaned by the initial metadata load (in a typical run, they aren't). It's also suprising that the reloads (via reloadOrphanedFiles) do not correctly result in the requisite file associations. Logging in those test failures indicates that the loaded packages include the orphaned files, yet they continue to be reloaded (EDIT: my eyes deceived me, the paths don't match! See below).

This is likely related to my recent changes to metadata graph building. I'd like to understand this before releasing gopls@v0.9.0. Unfortunately, it seems exceedingly rare; I will see if I can add more instrumentation and reproduce via slowbots.

CC @adonovan

[findleyr]

Ok, two new failures just occurred on darwin builders:
https://build.golang.org/log/2a5d73ac2f31f441c1b3a939c5c29addd21f9c1c
https://build.golang.org/log/438b80942ea9ff3a766b43d9fe90516aed97e69a

All things considered this is good news, because hopefully additional instrumentation will allow root causing. Unfortunately, this means this really must block the v0.9.0 release; unless we convince ourselves that there is something incorrect about the setup of internal/lsp tests, this failure rate is certainly high enough to affect our users.

The increased frequency also makes it more likely that a recent change contributed to these failures.

[gopherbot]

Change https://go.dev/cl/414854 mentions this issue: internal/lsp: add additional instrumentation around package loading

[findleyr]

Aha! I only just now noticed the asymmetry between query and result, in the test logs

2022/06/28 11:27:39 go/packages.Load
    snapshot=1
    directory=/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/tmp/gopls-workspace-mod3193680098
    query=[file=/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/tmp/TestLSP_MultiModule32642401/lsp/testmodule/interfacerank/interface_rank.go]
    packages=1
2022/06/28 11:27:39 go/packages.Load
    snapshot=1
    package="golang.org/x/tools/internal/lsp/interfacerank"
    files=[/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/tmp/TestLSP_MultiModule32642401/lsp/testmodule/testmodule/interfacerank/interface_rank.go]

Note that we're querying for testmodule/interfacerank/interface_rank.go, but getting back a package containing an extra 'testmodule': testmodule/testmodule/interfacerank/interface_rank.go. This would explain why it only affects the 'MultiModule' run mode, which creates this synthetic module.

Because the load succeeds, we don't mark these files as unloadable and continue to reload them ad-infinitum.

The bug may be in go/packages, or may be in one of the transformations gopls applies. I haven't figured out where the extra 'testmodule' is coming from.

[findleyr]

Ok, I've stared at this for a while, reading the gopls, go/packages, go list, and modindex code. With that said, I haven't made much progress.

This appears to be a go list or go/packages bug, perhaps related to the recent change to package indexing? The first failure was shortly after https://go.dev/cl/413815 landed ("cmd/go: add per-package indexing for modules outside mod cache", though I don't necessarily think that CL is implicated).

A few notes:

• the go.mod file starts out in lsp/go.mod but is moved to lsp/testmodule/go.mod, so perhaps that move could cause the extra "testmodule" due to improper invalidation. That seems plausible, but unlikely because the file move happens before we start up the tests, and therefore should be before we invoke the go command for the first time
• there are many packages in the module, but only a subset get this bug, which is strange
• go/packages doesn't appear to verify that GoFile paths match the file=query, only checking basenames. That would allow this bug: https://cs.opensource.google/go/x/tools/+/master:go/packages/golist.go;l=332;drc=7743d1d949f1006ad12b190d68996acafc84d1d6

@bcmills does this type of corruption ring any bells?

[rsc]

I am looking into modindex to understand a performance problem and will look into this behavior too.

[bcmills]

I wonder if the per-package index needs to include the corresponding module's go.mod file (contents and location) in the cache key. That could potentially affect the import path and the directory relative to the module root.

[gopherbot]

Change https://go.dev/cl/415475 mentions this issue: cmd/go: include module root in package index key