proposal: cmd/go: include the raw import path in the output of `go list -json`

[bcmills]

Background

The ImportPath field in the output of go list -json is not quite what it says on the tin. Instead of a package import path, it contains an import path augmented with a suffix for uniqueness.

This has led to several bugs and some confusion in the past:

• cmd/go: go test cannot find module path in module root directory #26722
• runtime: ExampleFrames with -trimpath fails #43380
• cmd/go: TOOLEXEC_IMPORTPATH is ambiguous for test packages #44963
• cmd/go: import paths can contain the .test suffix, breaking the toolchain #60454

It also adds complexity in tools that deal with the output of go list. For example, golang.org/x/tools/go/packages renames the field to ID and parses out the package path in an ad-hoc manner:
https://cs.opensource.google/go/x/tools/+/master:go/packages/golist.go;l=544-549;drc=f60f2e6aa42c945111053771ea53938694791d83

Since the way one interacts with the go command is by passing it package import paths, the need for this kind of ad-hoc parsing requires that users building tools on top of cmd/go must implement their own ad-hoc parsing, with varying degrees of accuracy.

Unfortunately, too many tools already expect the ImportPath field to contain this extra information. We cannot feasibly rename ImportPath itself to be clearer.

Proposal

I propose that we add a new field to the Package struct format produced by go list, containing the original, unmodified path by which the package's source code was located (in GOROOT, GOPATH, or the module cache).

I suggest that we name the new field PackagePath, along the lines of the existing PkgPath field reported by x/tools/go/packages.

Specifically:

• The path should not contain the suffix .test or _test unless that suffix is literally included in the directory or module path containing the package source code.
• The path should not contain suffixes to denote test or PGO variants.
• As a result, multiple packages reported by go list may share the same PackagePath. They can be distinguished using the (misnamed) ImportPath field, whose semantics remain unchanged.

[adonovan]

I suggest that we name the new field PackagePath, along the lines of the existing PkgPath field reported by x/tools/go/packages.

I was hoping this would allow us to simplify the go/packages logic for inferring the package path, but there's surprisingly little of it: with strings.Cut(" "), it would amount to a 1-liner.

Seems reasonable though. The name is consistent with the names we use for this concept elsewhere. ImportPath should ideally be ID, matching the concept used in go/packages, but as you point out, we can't change it now.

[mvdan]

Unfortunately, too many tools already expect the ImportPath field to contain this extra information. We cannot feasibly rename ImportPath itself to be clearer.

I assume we will carefully document both ImportPath and PackagePath fields in the docs, though :) I think that's a pretty important part of this proposal, actually.

We can even consider duplicating the new ImportPath field as ID , deprecating the former and discouraging its use. We can likely never remove it, and it would be duplication, but at least JSON users could use the very clear pair of fields ID and PackagePath rather than the pair ImportPath and PackagePath, which will always confuse the reader until they read the docs.