Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: store package metadata in build cache #31417

jayconrod opened this issue Apr 11, 2019 · 0 comments

cmd/go: store package metadata in build cache #31417

jayconrod opened this issue Apr 11, 2019 · 0 comments
GoCommand NeedsInvestigation Performance


Copy link

@jayconrod jayconrod commented Apr 11, 2019

The Go command currently spends a lot of time gathering package metadata using go/build.Import and cmd/go/internal/imports.ScanDir. Both functions read and parse .go files in directories in order to gather imports and evaluate build constraints. This is expensive both in I/O and CPU time. Source files rarely change, especially in the module cache.

We should investigate saving the package metadata we gather in the build cache. The cache key for a package would be derived from readdir on a package directory and stat on each file in the directory. If a file has changed, the stat metadata will almost certainly change as well: the modification time will be different, probably also the size.

We should take care to avoid hazards with mtime comparison. In particular, we should avoid caching stat data that is only a few seconds old because some file systems have low time resolution, and there is a window where several changes can be made without changing the recorded mtime. We may want to enable this only on certain file systems, operating systems, or only for the module cache.

An initial prototype (CL 75295) shows we can reduce go list time by roughly 20%. That may not be a big enough difference to justify added complexity, but profiling shows that more improvement may be possible if we use the cache for module loading and optimize deserialization.

@jayconrod jayconrod added this to the Go1.13 milestone Apr 11, 2019
@jayconrod jayconrod self-assigned this Apr 11, 2019
@andybons andybons removed this from the Go1.13 milestone Jul 8, 2019
@andybons andybons added this to the Go1.14 milestone Jul 8, 2019
@rsc rsc removed this from the Go1.14 milestone Oct 9, 2019
@rsc rsc added this to the Backlog milestone Oct 9, 2019
@jayconrod jayconrod removed their assignment Oct 14, 2021
@toothrot toothrot added the NeedsInvestigation label Oct 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
GoCommand NeedsInvestigation Performance
None yet

No branches or pull requests

4 participants