proposal: cmd/go: add FileHashes to go list -json output

[findleyr]

I think this is one of perhaps a few missing features that can unlock greater reliability in go<->gopls interop.

Background

Tools that consume go list -json (notably gopls, via go/packages) receive file lists (GoFiles, CompiledGoFiles, etc.) but have no way to know what the go command actually saw in those files. Between the time go list reads a file and the time the tool reads it, the content may change. This makes it impossible to detect whether information derived from go list -export or -compiled is coherent with the source the tool subsequently parses.

gopls in particular wants to detect this incoherence so it can re-invoke the go command rather than present stale or inconsistent results.

The existing BuildID field (#37281) is a related package-level witness, but it is salted with the toolchain version and folds in flags and transitive dependencies, so it changes when source files do not; it also requires -export. A per-file, unsalted content hash is the missing complement.

Proposal

Prototype in CL 768880.

Add a new field to the Package struct reported by go list:

// FileHashes maps each source file listed above to the hex-encoded
// SHA-256 hash of its contents at the time go list read it.
FileHashes map[string]string

The map key is the file name exactly as it appears in the corresponding file list (relative to Dir for source files, absolute for generated CompiledGoFiles). The value is the lowercase hex encoding of the SHA-256 digest. Files that cannot be opened are omitted.

The field is populated only when explicitly requested via -json=FileHashes (or referenced from -f). It covers all files reported in the source-file lists, including test and ignored files.

When -overlay is in effect, the hash is of the overlay content.

Cost

When go list is invoked with -compiled, -export, or a request for Stale, the build action graph already computes the SHA-256 of every input file (cmd/go/internal/work.buildActionID, via cache.FileHash) to form the cache key. In those modes—precisely the modes go/packages uses when type information is needed—reporting FileHashes adds no additional I/O or hashing; it is a map lookup per file.

When none of those flags is set, requesting FileHashes requires reading each file in full (the package loader otherwise reads only the file header through the import block). This is an opt-in cost paid only by callers that ask for the field.

Compatibility

This adds a field; existing consumers are unaffected. The -json=field,... selector means the field is never computed or emitted unless requested.

CC @matloob @adonovan

[adonovan]

I like this proposal. We've wanted it for a long time. However, before gopls can use these hashes, x/tools/go/packages will also need changes to plumb them through. (I wonder whether bazel, buck and other build systems are capable of computing them.)

The field is populated only when explicitly requested via -json=FileHashes

If go list is already computing all these hashes anyway, might it make sense to populate this field unconditionally? (That said, I think go list now computes hashes only when -json is requested and the Stale{,Reason} fields are requested; see https://go.dev/cl/740901 and #77419.)

[findleyr]

@adonovan there are some details to sort out, like do we want this for non-build files such as IgnoredFiles (which would incur an additional cost), but the fact that this was a 40 line change to the Go command, and would work around a whole class of gopls bugs, was a real head-slapper.

Do you want to bundle the go/packages changes with this proposal, since we really only want this for the corresponding change in go/packages?

[adonovan]

@adonovan there are some details to sort out, like do we want this for non-build files such as IgnoredFiles (which would incur an additional cost), but the fact that this was a 40 line change to the Go command, and would work around a whole class of gopls bugs, was a real head-slapper.

Especially since we'd hypothesized this very class of bugs and described the solution years ago! FWIW Herb's -json output emitted exactly this information for exactly this reason and that was in 2018.

Do you want to bundle the go/packages changes with this proposal, since we really only want this for the corresponding change in go/packages?

Yes, let's do that. I think the API should be specified the same way (i.e. map of file names exactly as they appear), but of course the file names in packages.Package's file lists are absolute. The covered file lists should of course include GoFiles and CompiledGoFiles. What about OtherFiles, IgnoredFiles? They are used by some analyzers. I think EmbedFiles isn't needed because its content cannot affect type checking.

I think we should include Package.Module.GoMod too, in go list and go/packages.

[seankhliao]

is the idea that gopls could also calculate its own hash of its view of the file and reinvoke cmd/go when it's different?

[findleyr]

@seankhliao gopls already computes sha256 hashes of all the files it reads, and eagerly reads files immediately after packages.Load. In other words, both sides have the exact same coherency mechanism and were not transmitting it.

And yes, we'd re-load until we get a coherent result.