Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: load packages in parallel #29758

jayconrod opened this issue Jan 15, 2019 · 4 comments


None yet
4 participants
Copy link

commented Jan 15, 2019

Currently, when the Go command loads a set of packages and their dependencies, it does so on a single goroutine. Tracing shows that most cores are idle when loading packages for a large build, so there's significant performance opportunity here. On the one core that's busy, most of the time is spent in internal/load.LoadImport and internal/load.Package.load (which are mutually recursive).

One may expect the package loading process to be dominated by I/O, but the files loaded are usually in the kernel file cache, and we do a lot of work in userspace to parse files, extract imports, and apply build constraints. We actually spend very little time in the kernel, at least on Linux (system call overhead is very low).

Based on this information, it seems that we can significantly speed up the package loading process by parallelizing it. This will benefit go list in particular, which is heavily used by go build and other commands may see some improvement, too.

@jayconrod jayconrod added this to the Go1.13 milestone Jan 15, 2019

@jayconrod jayconrod self-assigned this Jan 15, 2019


This comment has been minimized.

Copy link
Contributor Author

commented Jan 15, 2019


This comment has been minimized.

Copy link

commented Jan 16, 2019

Also just to note Chatting with @heschik, ideally go list -find would be made sufficiently fast to not require dropping down to go list -m and re-building package discovery logic atop that.

So hopefully this change will go a good step towards that goal.


This comment has been minimized.

Copy link

commented Feb 6, 2019

Change mentions this issue: cmd/go: parallelize package loading


This comment has been minimized.

Copy link

commented Mar 15, 2019

Change mentions this issue: cmd/go: refactor load.LoadPackage into other functions

gopherbot pushed a commit that referenced this issue Apr 1, 2019

cmd/go: refactor load.LoadPackage into other functions
LoadPackage was used to load a *load.Package for a command line
argument, after pattern expansion. It provided two special cases on
top of LoadImport. First, it ensured that "cmd/" packages in GOROOT
were installed in "$GOROOT/bin" or "$GOROOT/pkg/tool". Second, it
translated absolute paths to packages in GOROOT and GOPATH into
regular import paths.

With this change, LoadImport now ensures "cmd/" packages have the
right Target (without the need for a special case) and
search.ImportPaths translates absolute paths.

LoadPackage no longer handles these special cases and has been renamed
to LoadImportWithFlags, since it's still useful for loading implicit

Updates #29758

Change-Id: I9d54036f90c3ccd9b3a0fe0eaddaa7749593cc91
Run-TryBot: Jay Conrod <>
TryBot-Result: Gobot Gobot <>
Reviewed-by: Bryan C. Mills <>

@gopherbot gopherbot closed this in cebc4e5 Apr 5, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.