proposal: go/types: allow (*Package).Imports to be precise for imported packages

[mdempsky]

The documentation for go/types.(*Package).Imports says:

Imports returns the list of packages directly imported by pkg; the list is in source order.

If pkg was loaded from export data, Imports includes packages that provide package-level objects referenced by pkg. This may be more or less than the set of packages directly imported by pkg's source code.

This proposal is to change the second paragraph from saying "Imports includes packages" to "Imports may instead return packages".

--

Rationale: The go/types type checker's semantics of reporting the actual packages that were directly imported seems the clearly preferable behavior for end users. The alternative behavior when a package is read from export data and Imports instead returns an unspecified subset of transitive dependencies seems to always be less useful.

The reason for the current behavior is that cmd/compile's existing export data formats (textual, binary, and then indexed) were designed for the compiler's own needs, which do not require serializing a precise representation of the import graph. So instead the go/types importer approximates this by setting the Imports list to all packages seen in the export data.

For unified IR, the new export data format takes a go/types-first approach, and so now it precisely serializes the import graph. However, this technically violates the current semantics codified in the documentation. It's my position that the documentation was wrong to mandate that behavior, rather than document it as a known limitation.

If users write code that correctly walks the transitive dependency graph themselves, then their code will work both with packages created by go/types (e.g., using the srcimporter implementation) or imported from compiler export data (modulo cases where packages are omitted due to not providing symbols relevant to export data).

Finally, I'll note that the current set of packages included in Imports is unstable across compiler changes, because it includes not only packages referenced by the package's exported API (visible through go/types), but also packages that are needed by inline bodies (which are opaque to go/types importers). The meaning of "package-level objects referenced by pkg" seems too imprecise for users to rely on it meaning any particular set of packages.

--

Aside: It's also the case that the existing gcimporter implementations only call SetImports on the package being directly imported, not any transitively imported packages. Also, gccgoimporter never calls SetImports. I'm not sure whether to attempt codifying/fixing these behaviors. E.g., "If pkg is incomplete, Imports may instead return nil." would seem reasonable to allow gcimporter's existing behavior, and then gccgoimporter could be updated to implement the same imported-packages list approximation as gcimporter.

[gopherbot]

Change https://go.dev/cl/419596 mentions this issue: [dev.unified] go/internal/gcimporter: flatten imports

[ianlancetaylor]

CC @findleyr @adonovan

[rsc]

Didn't changing this break a test inside Google until we rolled it back?
I'm reluctant to break external users too.

We could add a (*Package).DirectImports of course.

[mdempsky]

Didn't changing this break a test inside Google until we rolled it back?

It caused a Google-internal documentation generator tool to start failing, yes. The team approved a change to make the tool walk the transitive dependency graph itself.

We haven't investigated yet if there are other tools depending upon the same behavior.

[findleyr]

If I'm understanding correctly, the current behavior is somewhat ill-defined but could still reasonably be relied upon, because the result of Imports is always a super-set of packages reachable from export data. Note also that the documentation says "Imports includes", not "Imports is equal to", so the fact that there may be additional packages due to inlining does contradict the definition. I'm leaning toward not changing the current behavior, based on the fact that doing so could (and did) break usage that was not incorrect.

Furthermore, even the new definition is not as useful as it could be, because Imports still means different things depending on whether the package came from export data (and we can't change that, because legacy importers will still exist). An advantage of a new API like DirectImports is that it can have an unambiguous meaning.

I don't feel strongly though. This does fall into a gray area, and it may be better to endure a small amount of breakage to avoid API cruft.

Aside: if we don't already, we should expose an API in x/tools that provides a more accurate set of packages reachable from export data. Users should not rely on the current behavior.

[mdempsky]

Note that even if we add a DirectImports method, legacy importers still won't be able to implement it, because the export data formats themselves lack the necessary information.

[findleyr]

Note that even if we add a DirectImports method, legacy importers still won't be able to implement it, because the export data formats themselves lack the necessary information.

Understood. But that may be an easier limitation to document.

[griesemer]

Currently, the result of Packages.Imports for packages loaded from export data is not really well-defined as @mdempsky has pointed out - it depends on the compiler implementation. Any client of go/types that is relying on Packages.Import will need to use a source-based importer for accurate/usable results such as a proper import tree, etc. - the current set of imported packages listed on packages imported from compiler-export data is not particularly useful.

I'd be helpful to hear from clients of go/types that make use of Packages.Imports that have contrary evidence.

As is, I am mildly in favor of adjusting the documentation as suggested.