Implement a clustermq backend with persistent workers

[wlandau]

https://github.com/mschubert/clustermq. Looks promising for fast transient workers.

[wlandau]

I just tried clustermq on SGE, and workers spin up super fast! I am eager to develop clustermq-based backends for both persistent and transient workers. May be awhile before I can make the time, though.

[wlandau]

From #431, just as I thought, clustermq does not solve the target-level overhead problems for persistent workers. The best way to use clustermq is transient workers with caching on the master process.

[kendonB]

As I've mentioned in another issue, caching on the master process is going to be slow for I/O heavy jobs. I've seen speed improvements on I/O heavy jobs on a GPFS file system using up to about 200 simultaneous workers. If those jobs were caching using a single process, it wouldn't be much better than doing the compute stage with a single process.

This is not to discourage you from pursuing clustermq, just to consider having an option to having caching done by the workers.

[wlandau]

Sure, I understand. I am thinking clustermq is be ideal for situations with large numbers of small targets, where the overhead of launching remote jobs and accessing NFS is the bottleneck.

[kendonB]

One thing for the todo here would be to expose the template argument to clustermq::workers

[wlandau]

Re #425 (comment): great idea, @kendonB. I did not have enough bandwidth to address this detail back in July, but I just added it in 3288dc5.