Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
cmd/go: load packages in parallel #29758
Currently, when the Go command loads a set of packages and their dependencies, it does so on a single goroutine. Tracing shows that most cores are idle when loading packages for a large build, so there's significant performance opportunity here. On the one core that's busy, most of the time is spent in
One may expect the package loading process to be dominated by I/O, but the files loaded are usually in the kernel file cache, and we do a lot of work in userspace to parse files, extract imports, and apply build constraints. We actually spend very little time in the kernel, at least on Linux (system call overhead is very low).
Based on this information, it seems that we can significantly speed up the package loading process by parallelizing it. This will benefit
Also just to note https://go-review.googlesource.com/c/tools/+/158097. Chatting with @heschik, ideally
So hopefully this change will go a good step towards that goal.
LoadPackage was used to load a *load.Package for a command line argument, after pattern expansion. It provided two special cases on top of LoadImport. First, it ensured that "cmd/" packages in GOROOT were installed in "$GOROOT/bin" or "$GOROOT/pkg/tool". Second, it translated absolute paths to packages in GOROOT and GOPATH into regular import paths. With this change, LoadImport now ensures "cmd/" packages have the right Target (without the need for a special case) and search.ImportPaths translates absolute paths. LoadPackage no longer handles these special cases and has been renamed to LoadImportWithFlags, since it's still useful for loading implicit dependencies. Updates #29758 Change-Id: I9d54036f90c3ccd9b3a0fe0eaddaa7749593cc91 Reviewed-on: https://go-review.googlesource.com/c/go/+/167748 Run-TryBot: Jay Conrod <firstname.lastname@example.org> TryBot-Result: Gobot Gobot <email@example.com> Reviewed-by: Bryan C. Mills <firstname.lastname@example.org>