Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
cmd/go: build dependent packages as soon as export data is ready #15734
This is a trace of the activity on an 8 core machine running 'go build -a std':
For those who want to explore more, here is an html version. (Hint: Use a, d, w, and s keys to navigate.)
There are a few early bottlenecks (
We currently wait until a package is finished building before building packages that depend on it. However, dependent packages only need export data, not machine code, to start building. I believe that that could be available once we're done with escape analysis and closure transformation, and before we run
For the bottlenecks listed above:
Though slightly optimistic (writing export data isn't instantaneous), this does suggest that this would in general significantly reduce time spent waiting for dependencies to compile.
This pattern of large, slow, linear dependency chains also shows up in bigger projects, like juju.
@rsc implemented one enabling piece by adding a flag to emit export data separately from machine code.
Remaining work to implement this, and open questions:
Given the scope of the change, I'm marking this as a proposal. I'd love feedback.
This was referenced
May 18, 2016
Love it. Also glad that traceview ended up working out.
Localhost TCP? Later: multi-machine TCP and making cmd/compile and cmd/link use a VFS (and network impls) rather than the os package for file access. Imagining running a large build on a cloud machine and cmd/go spinning up some helper Kubernetes (or whatever) containers to speed the build, then going away when the build is done, paying for them by the number of seconds they were running.
A relatively simple (but local-only) way would be:
I also suspect you don't actually need separate export data file functionality for this. Since the export data is at the beginning of the .a file, we can just write a partial .a file, signal cmd/go, and then finish writing later.
Interesting. I suspect also that the time from start of compilation until the time the export data becomes available will become smaller over time (faster frontend), while the backend may become slower (relatively), due to more powerful optimizations. Seems like a good idea to me.
Only vaguely connected random idea: If the export data hasn't changed, can
On 19 May 2016 at 10:16, Robert Griesemer email@example.com wrote:
referenced this issue
May 19, 2016
changed the title
proposal: build dependent packages as soon as export data is ready
Aug 22, 2016
referenced this issue
Apr 21, 2017
I'm very skeptical this is worth the complexity. It would require support for "half-completed" actions in the go command where the compile step half-completes early and then fully-completes later. I don't believe the payoff here would be worth the significant increase in complexity. I guess you could run two compiles, so that you generate the export data first and then the whole object second, but that just does more work overall. Even if it improves latency in certain cases, more work overall is a net loss.
Critical path scheduling or just working on making the compiler faster seems like a better use of time.