Add MPI-capable tasks

[ocaisa]

(in the same namespace)
Fixes #8

I'm not 100% sure about how I approach the serialization is the right way. Do you know what the cost of serialization is? Here I effectively do it three times, I serialise the function, dask serialises the result, MPI serialises the result of that (to do the broadcast).

The restriction here is that only the root MPI process can return something back to dask (so any management related to that has to be handled by the task itself).

[AdamWlodarczyk]

Overall, I really like this approach. if this solution shall work only for mpi tasks which are essentially primitive this should work flawlessly (I am pointing at problems which @dwhswenson and me had some time age with serialization of OPS storage objects).
Since some validation in (de)serialization I would go for creation some class which would handle most of that stuff. May I suggest some class-based solution?

[ocaisa]

@AdamWlodarczyk The serialization stuff is my weak point, if you can suggest something that would be great.

[ocaisa]

What's left is to swap out the _command_template of https://github.com/dask/dask-jobqueue/blob/master/dask_jobqueue/core.py#L284 replacing distributed.cli.dask_worker with jobqueue_features.mpi_dask_worker and then wrap the whole command using mpi_wrap

[ocaisa]

@AdamWlodarczyk This is actually working now. Can you review? I'd like to cut a release of this before our workshop. I'll add tests and maybe refactor a bit next week.

[AdamWlodarczyk]

It looks fine for me.

[ocaisa]

It looks fine for me.

Thanks for talking a look @AdamWlodarczyk I'll address that comment and add some additional testing and then merge this so I can cut a release to install on JURECA.

I plan on reworking the whole test process to emulate what dask-jobqueue does so that we can actually test all these features in the CI (but that is for a separate PR).