POC [Fixes #311] fix cleanup

[stefanpenner]

• ensure cleanup gives the ability to wait until it is done
• improves support for mid-build cleanup, waits for the current task to complete, prevents the next task, and then exists.

[joliss]

I don't think it's correct to try removing tmp dirs with builderTmpDirCleanup while the build is still in progress. We'd need to wait for the current step to finish first.

[joliss]

Oh. Wait, I think you're already saying this in your original comment (your list is a to-do list, yes?)...

[stefanpenner]

ya, the version in broccoli-builder 0.18.x waits

[joliss]

Tests will catch this anyway, but there's a tyop here (self._canceled) :)

[joliss]

I actually wonder if it's conceptually necessary to tie the cancellation thing to the cleanup function. In other words, I don't think there's any reason why in general we couldn't start a fresh build after canceling the current build, instead of cleaning up immediately. We shouldn't end up in any "wedged" state that would stop us from doing so.

So if we do that, the Ctrl+C handler would do something like:

cancelCurrentBuild()
then wait for currentBuild
then cleanup

[stefanpenner]

@joliss i'll update this to be more like what I ended-up with on 0.18.x. There are some nuanced differences, that have not made it to this PR yet.

---

Some thoughts:

• adding a public cancel independent of cleanup is totally fine, will gladly add that. We can also make an non exceptional cancel recoverable.
• cleanup should likely still call cancel, otherwise cleanup and an running build will simply race. We can also make a non exceptional cleanup recoverable
• we need to decide the ordering between the completion of cancel() cleanup() and builder.build(). I believe builder.build completion must wait for any pending cleanup() which in tern must wait for any pending cancel(). That way the external world remains observably consistent, regardless if a build completes via rejection due to cancellation, due to error, or totally normal runtime.
• the system should not allow any more then 1 build, 1 cleanup and 1 cancel at any point in time, and they must correctly wait on each other to ensure the build remains observably consistent.

[stefanpenner]

replaced by: #350