Speed up the compilation process v5

[ghost]

Sames as #113 except:

• trivial fixes and updates
• build info data refactoring
• using term_to_binary compression
• not writing build info file if the graph is unmodified
• default enabling keep_build_info
• unconditionally keeping build info
• making the info file universal and storing it as <base_dir>/.rebarinfo
• introducing <base_dir>/.rebar/ and storing erlc build info there
• fixing regressions

[proger]

I've been using eflame to profile rebar runtimes in cases when all files already compiled (so ideally rebar should finish instantly), but it seems like init_rebarinfo is taking some time to gather infos when it's not necessary — is it possible to make it not do that?

(the way to build this picture is explained at the bottom of eflame's README)

[ferd]

Regarding init_rebarinfo, I'm guessing the biggest gains in time could be to confirm whether the graph has been modified or not on a run, and then avoiding re-writing the file when that happens.

Right now it looks like the entire project is serialized again every time, and flushed back to disk, even if nothing changed.

That would be a low hanging fruit, I guess.

[dizzyd]

@ferd @Tuncer I'm not going to review all 5 versions of this PR - can we please pick one and close the others?

[ferd]

@dizzyd my understanding is that each branch does more than the other. You could review v5 only, and if you don't like a feature or something, it may turn out that v4, v3, v2, or v1 already have the separation for that.

[ghost]

Regarding init_rebarinfo, I'm guessing the biggest gains in time could be to confirm whether the graph has been modified or not on a run, and then avoiding re-writing the file when that happens.

Right now it looks like the entire project is serialized again every time, and flushed back to disk, even if nothing changed.

That would be a low hanging fruit, I guess.

@ferd, @proger, any technical issue with postponing that to after we merge one of the 5 variants first?

[ferd]

No issue. It's only an inefficient thing. It can be fixed without
breaking anything also.

On 09/20, Tuncer Ayaz wrote:

Regarding init_rebarinfo, I'm guessing the biggest gains in time could be to confirm whether the graph has been modified or not on a run, and then avoiding re-writing the file when that happens.

Right now it looks like the entire project is serialized again every time, and flushed back to disk, even if nothing changed.

That would be a low hanging fruit, I guess.

@ferd, any technical issue with postponing that to after we merge one of the 5 variants first?

---

Reply to this email directly or view it on GitHub:
#129 (comment)

[ghost]

@proger, @ferd, thanks for the init_rebarinfo suggestion. I made the necessary changes.

[archaelus]

I've started using this version of the patch day-to-day and I like it. Rebar seems super fast now.

👍

[tsloughter]

I've been using it for a while now. No issues.

[jaredmorrow]

Since this also adds the notion of a new file, I will tag rebar as it is, then test this branch as well as I can and merge it in, any problems with that @Tuncer?

[ghost]

Works for me.

[ghost]

While discussing lock-deps and erlc-speedup-v5 with @seth, he suggested to introduce a directory for all info files. That's a good idea, and with that implemented, I consider erlc-speedup-v5 ready to merge. That directory is also a good place to store a rebar lock-deps info file.

[archaelus]

We've been using this patch on our version of rebar at Heroku and it kicks ass - compilation speed is fantastic and day-to-day development feels much better with this patch in rebar.

💯 I would love to see this patch land in mainline.

[drobakowski]

Hi @Tuncer and @zinid,

thanks a lot for the great speed improvements! I’ve tried this patch for some projects but I’ve encountered some compilation errors while using it. One project has a behaviour module located in a subfolder of src/ e.g. src/behaviour/some_behaviour.erl. You can easily reproduce the error for example with poolboy:

mkdir src/behaviour && mv src/poolboy_worker.erl src/behaviour/ && cp test/poolboy_test_worker.erl src/

Errors while trying to compile:

==> poolboy (compile)
Compiling src/poolboy_test_worker.erl failed:
src/poolboy_test_worker.erl:3: behaviour poolboy_worker undefined
ERROR: compile failed while processing /private/tmp/poolboy: rebar_abort

Another error appears while trying to compile erlando:

==> erlando (compile)
Compiling src/monad_plus.erl failed:
src/monad_plus.erl:none: undefined parse transform 'do'
ERROR: compile failed while processing /private/tmp/erlando: rebar_abort

Bisecting with git located the first bad commit at 0796ffe, for both projects.

[ghost]

@drobakowski thanks for the bug report. I'll look into it.

[ghost]

@drobakowski can you please retest?

[drobakowski]

@Tuncer compiling erlando now works fine! The first example with the modified poolboy worker still doesn't compile. I've uploaded you the modified poolboy worker src on DB.

[ghost]

@drobakowski that's right, I didn't fix that yet.

[Vagabond]

This variable name seems to have a bit of redundancy...

[ghost]

Renamed to ErlFirstFiles

[Vagabond]

If this can be squashed into a saner commit history, I think this is ready to merge.

[Vagabond]

As some anecdotal evidence, it takes a riak rebuild from 20 to 10 seconds and riak_cs build from 10 to 5 seconds, which is very nice for iterative development.

[ghost]

Before we can merge this, I first need to fix the 2nd bug reported by @drobakowski, and I'd also welcome a review of the code that skips updating the info file if there's no change. Specifically, whether the logic is correct for the case of a compile error, and if in that case a different update strategy is needed.

Once that's all done, I'll look at the commit history.

[Vagabond]

The test case provided by @drobakowski passed for me, when I tried it. I thought you had fixed it.

[ghost]

I have only fixed the first bug. The second problem happens in the 'modified poolboy worker' scenario.

[Vagabond]

I can't replicate the error, but maybe I don't understand how to.

[drobakowski]

@Tuncer the compile errors also occurs with -j1 and also with i386. Building the erlang vm with halfword-emu doesn't seem to work quiet well on a Mac and hence I'm unable to test it with this vm. Sorry, maybe someone else could try to reproduce this with halfword-emu?

[drobakowski]

I'm able to reproduce the error on Ubuntu Server 12.04.4 LTS (GNU/Linux 3.2.0-27-generic x86_64) with R16B03-1 64-bit halfword-emu.

[Vagabond]

FWIW I am on Arch Linux x64 with erlang R16B02.

Andrew

[ghost]

@drobakowski I have uncovered a 2nd bug and fixed both regressions. Can you please retest?

[drobakowski]

@Tuncer great, 1c8a9ae and 5e81289 are working fine now but fixing "catch all" in 70ce087 lead to some other error. See the diff note.

[ghost]

@drobakowski thanks for testing, backed out 70ce087. I'd prefer to not hide all erl_syntax[_lib] errors and replaced it with a TODO comment.

[drobakowski]

@Tuncer thanks a lot for your great work! :) 👍

[ghost]

ping

[archaelus]

This is wonderful in day-to-day use - are there any blockers to merging it into mainline?

[jaredmorrow]

No, I was originally waiting on the bugs @drobakowski found, and hadn't gotten back around to checking in.

[Vagabond]

Can you squash this into a saner commit history, please?

[Vagabond]

Were the two bugs holding things up fixed here: https://github.com/tuncer/rebar/commit/44ff85aba3a05ad78bd93e18907b2598259a15c3 ?

Also, you fixed the bugs, but this branch adds nothing to the regression tests? Should we have tests for those edge cases?

[ghost]

Can you squash this into a saner commit history, please?

I could probably squash a8372f7 0303434 88dcb6f 9369311 92be8c1 31ebeb1, but I'm not sure we'd want to collapse all the regression fixes and behavioral changes into one or two commits. If that's sufficient, I'll make the changes tomorrow.

[Vagabond]

Are they regressions that affected rebar before this branch, or were they regressions added in the branch? I'm fine with >1 commits for this, but 26 is pretty far beyond what I'd expect for a change of this size. It isn't even easy to tie all these back to this feature being added, if someone was to do a git blame.

[ghost]

Were the two bugs holding things up fixed here: tuncer@44ff85a ?

Yes.

Also, you fixed the bugs, but this branch adds nothing to the regression tests? Should we have tests for those edge cases?

We don't really have a test suite for rebar_erlc_compiler, but we should create one.

[Vagabond]

Or we can add an inttest that mirrors the issue.

[ghost]

Are they regressions that affected rebar before this branch, or were they regressions added in the branch?

New regressions added by the initial commit.

I'm fine with >1 commits for this, but 26 is pretty far beyond what I'd expect for a change of this size. It isn't even easy to tie all these back to this feature being added, if someone was to do a git blame.

If you want me to collapse it into a couple commits, I can try to do that, but it would be a couple commits with long commit messages. Keeping the various fixes and behavioral changes separate helps if we have to git-bisect a problem.

Anyway, I'll see what I can squash.

[Vagabond]

I'd rather make git bisect and git blame work better than have so many commits.

[ghost]

Or we can add an inttest that mirrors the issue.

That's what I meant by test suite. I'm not sure when I'll have time for that. So, if anyone wants to speed up the merge process, here's what inttest/erlc should most most likely do:

• single demo project that uses all features and file types supported by rebar_erlc_compiler and also requires non-trivial compile priority to successfully build
• initial erl_opts should not have no_debug_info
• run rebar compile
• verify result
• add no_debug_info to erl_opts
• run rebar clean compile
• verify that beam files have no debug info

[ghost]

I'd rather make git bisect and git blame work better than have so many commits.

Collapsed commits as much as possible.

[Vagabond]

Thank you. I will look at this today.

[ghost]

Test suite included in #242.