x/tools/gopls: detect and recover from snapshot/disk incoherence

[findleyr]

A snapshot caches the content of every file it has read, to keep derived calculations such as type-checked packages coherent. But go list reads disk, not the cache, so the two can diverge whenever a closed file changes on disk without a corresponding didChangeWatchedFiles event. #64235 is one symptom (repro and mechanism); #63822 is probably another.

In other words, we hold every workspace file in memory to defend against incoherence, and it isn't sufficient: with the current go command APIs there is no way to guarantee that go list and gopls observe the same file state. We should stop paying the memory cost for a guarantee we don't get, and instead make incoherence recoverable.

Concretely:

• Detect the inconsistency by checking for inconsistent metadata from Load, or unexpected file content changes. It would be nice if the go command returned a manifest of file identities for files it read.
• Recover by recreating the view from scratch; we've so heavily optimized start up time so that this should be possible.
• Degrade gracefully by making most things work when gopls is initializing or reinitializing (x/tools/gopls: long initial workspace load durations for workspace with many packages #69523) -- I'm doing this.
• Alert the user. Mechanism T.B.D., a "reinitializing" progress at least, but maybe also a popup warning of incoherent disk state.
• Drop file contents 🥳

Alternatives:

• copy all the files to a temp directory and run go list there (this is effectively like passing a full overlay). 🤮

[gabyhelp]

Related Issues

• x/tools/gopls: invalidate snapshot when a new file is added to the package #34678 (closed)
• x/tools/gopls: watch file changes on disk #31553 (closed)
• x/tools/gopls: eliminate dependence on "seen files" #57558 (closed)
• x/tools/gopls: diff metadata before invalidating on changes to go.mod #36589
• x/tools/gopls: reinitialize snapshot eagerly and asynchronously #42365
• x/tools/gopls: retry workspace initialization as needed #36854 (closed)
• x/tools/gopls: improvements to loading behavior #68002
• x/tools/gopls: failure to recover from cancelled reinitialization #57626 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}