Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: only rebuild dependent packages when export data has changed #15752

Open
josharian opened this issue May 19, 2016 · 10 comments
Open

cmd/go: only rebuild dependent packages when export data has changed #15752

josharian opened this issue May 19, 2016 · 10 comments

Comments

@josharian
Copy link
Contributor

@josharian josharian commented May 19, 2016

@mwhudson suggested in #15734:

If the export data hasn't changed, can you skip the compilation of dependent packages?

Moving this idea here to be tracked separately.

My 2c: Interesting idea, seems worth pursuing. Not sure how often it happens that code changes don't impact export data (or how cheap that is to detect), but when it does happen, that could save a bunch of computation.

@dr2chase
Copy link
Contributor

@dr2chase dr2chase commented May 19, 2016

It's not trivial. Years a go I worked on an ahead-of-time compiler for Java bytecodes that did just this, and it was a good thing (back in the days of 200Mhz CPUs), but it required substantial amounts of dependence information (stored on disk in gzip format) and it also required that every reference to information from another package log an entry in the internal dependence database for later storage -- in particular, every field reference or interface method reference, because those involve offsets, and offsets might change as interface declarations or field declarations are changed (granularity matters, too -- do you record the actual dependence that field F is at offset 8, or do you just record that the SHA of the struct had a particular value?

Loading

@josharian
Copy link
Contributor Author

@josharian josharian commented May 19, 2016

IIUC, the plan would be to do something like check the SHA of the export data. If that hasn't changed, then the downstream compilation won't change.

Loading

@mwhudson
Copy link
Contributor

@mwhudson mwhudson commented May 19, 2016

Yeah, I was just thinking that you'd store the SHA1 of the export data of
all imported packages in the .a file and only recompile if any of them had
changed. No sense trying to be too sophisticated.

I don't know often this would be useful. Certainly sometimes (especially in
a "add more and more debug prints" cycle), but it will obviously also have
a small cost all the time. Seems worth trying out, I'll try to hack
something up between now and 1.8 early.

On 20 May 2016 at 06:56, Josh Bleecher Snyder notifications@github.com
wrote:

IIUC, the plan would be to do something like check the SHA of the export
data. If that hasn't changed, then the downstream compilation won't change.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#15752 (comment)

Loading

@griesemer
Copy link
Contributor

@griesemer griesemer commented Jun 2, 2016

Régis Crelier (currently at Google) has investigated this issue for Oberon in very much detail - I believe much of his work should be directly applicable to Go. For details see:
http://e-collection.library.ethz.ch/eserv/eth:39386/eth-39386-02.pdf

Loading

@josharian
Copy link
Contributor Author

@josharian josharian commented Jul 6, 2017

@rsc will this be a part of your cmd/go re-building improvements?

Loading

@rsc
Copy link
Contributor

@rsc rsc commented Oct 25, 2017

Yes the plan is that we'll be set up well to take advantage of this.
We probably won't do it initially, but as a followup.

Loading

@josharian
Copy link
Contributor Author

@josharian josharian commented Oct 9, 2018

Note that if we do much with #25999, the set of code changes that leave export data unchanged might shrink significantly.

Loading

@josharian
Copy link
Contributor Author

@josharian josharian commented Apr 30, 2019

cc @bcmills @jayconrod

Is this now just (“just”) a matter of only hashing the export data when determining whether object files are stale?

Loading

@bcmills
Copy link
Member

@bcmills bcmills commented Apr 30, 2019

I think so, yes.

Loading

@mwhudson
Copy link
Contributor

@mwhudson mwhudson commented Apr 30, 2019

I think it's a little bit more than that because you still need to re-link any executables?

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants