Add fault tolerance to scheduler

[jpsamaroo]

When a worker is killed due to OS signals (such as the Linux OOM
killer), a ProcessExitedException is likely to be thrown to the head
node. When such an exception is detected, we reschedule the failed thunk
on a new, non-dead worker (randomly for now).

Todo:

• Reschedule dependent thunks that were cached on the dead worker
• Recursively reschedule all dependent inputs of dead thunks if not cached on a live worker
• Fix various KeyErrors due to not getting state just right...
• Add some tests

[jpsamaroo]

@shashi @aviks please excuse the @debug statements, but are you otherwise content with the (locally passing) tests I've added to test/fault-tolerance.jl? I won't make the assertion that it will be able to correct every possible DAG and worker failure situation, but it seems to work properly for some common patterns.

This should also have no effect on normal (non-failure) situations, since the fault handling code only triggers when a worker dies unexpectedly.

[jpsamaroo]

Also, I will be moving most of this code out into its own file so that I don't make scheduler.jl hard to read.

[jpsamaroo]

bump

[jpsamaroo]

I also think it would be good to add an option to the scheduler to disable this new recovery functionality, in cases where it is undesired. I don't want to mess with this PR any more than I need to, so I will be doing that in a new PR soon (which will also expose some other unrelated options for the scheduler).

@shashi @aviks @tanmaykm any chance one of you can review this PR and let me know if there is anything else desired before merge?

[jpsamaroo]

I'm going to assume the single Travis failure on nightly is not my fault 🙂

[versipellis]

I tested this with the dataset that was giving me troubles on JuliaDB which spawned the initial convos around worker fault tolerance with @jpsamaroo, and can verify that it resolves the Linux OOM killer issue. There does seem to still be a memory leak either with JuliaDB or Dagger (unrelated to the fault tolerance) where about ~2gb of memory remains allocated after the processes have completed.

[shashi]

Going to review it tonight! This is a lot of code for this package haha.

[jpsamaroo]

Hi @shashi, did you get the chance to review this PR? I'm happy to answer any questions you may have about how it works, since not everything I included is obvious (or possibly even necessary) 🙂

[jpsamaroo]

@shashi @andreasnoack if there are no objections, I'm going to merge this.

[coveralls]

Coverage increased (+2.1%) to 54.15% when pulling 59e57bb on jpsamaroo:jps/fault-tolerance into 777b002 on JuliaParallel:master.

[jpsamaroo]

Thanks!

[aviks]

Thanks for the work here, @jpsamaroo . Does JuliaDB tests pass on this code? Would you please verify that before tagging a release here?

[jpsamaroo]

I had the exact same thought 🙂 I'm running tests locally as we speak. Are there any other packages that rely on Dagger that I should test? Also, maybe we should setup some reverse dependency testing in CI, since JuliaDB is a pretty important consumer of Dagger.

[jpsamaroo]

JuliaDB master is passing with Dagger master and MemPool latest release and master.