New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: load packages in parallel #29758

Open
jayconrod opened this Issue Jan 15, 2019 · 3 comments

Comments

Projects
None yet
4 participants
@jayconrod
Copy link
Contributor

jayconrod commented Jan 15, 2019

Currently, when the Go command loads a set of packages and their dependencies, it does so on a single goroutine. Tracing shows that most cores are idle when loading packages for a large build, so there's significant performance opportunity here. On the one core that's busy, most of the time is spent in internal/load.LoadImport and internal/load.Package.load (which are mutually recursive).

One may expect the package loading process to be dominated by I/O, but the files loaded are usually in the kernel file cache, and we do a lot of work in userspace to parse files, extract imports, and apply build constraints. We actually spend very little time in the kernel, at least on Linux (system call overhead is very low).

Based on this information, it seems that we can significantly speed up the package loading process by parallelizing it. This will benefit go list in particular, which is heavily used by golang.org/x/tools/go/packages. go build and other commands may see some improvement, too.

@jayconrod jayconrod added this to the Go1.13 milestone Jan 15, 2019

@jayconrod jayconrod self-assigned this Jan 15, 2019

@jayconrod

This comment has been minimized.

Copy link
Contributor Author

jayconrod commented Jan 15, 2019

@myitcv

This comment has been minimized.

Copy link
Member

myitcv commented Jan 16, 2019

Also just to note https://go-review.googlesource.com/c/tools/+/158097. Chatting with @heschik, ideally go list -find would be made sufficiently fast to not require dropping down to go list -m and re-building package discovery logic atop that.

So hopefully this change will go a good step towards that goal.

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 6, 2019

Change https://golang.org/cl/161397 mentions this issue: cmd/go: parallelize package loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment