Joblib interface

[mrocklin]

Should we support a joblib interface alongside the concurrent.futures interface? This would be for simple embarrassingly parallel computation. It would require us to think about auto-batching long sequences of small inputs to ensure we tailored batch size to hit a nice frequency of output. This would add novel capability to existing joblib users in two ways:

1. They could nest several functions that call out to joblib smoothly (this is a current pain point)
2. They would get distributed computation

This would be a nice way to support existing codebases within libraries like scikit-learn.

[ogrisel]

Now that the backend refactoring of joblib has been merged in master I would be in favor of shipping a distributed backend implementation instead.

Here is an example of scikit-learn using distributed via this backend:

https://github.com/ogrisel/docker-distributed/blob/master/examples/sklearn_parameter_search.ipynb

the code of the backend is there:

https://github.com/ogrisel/docker-distributed/blob/master/examples/distributed_joblib_backend.py

[mrocklin]

Was the joblib-distributed backend used in that notebook? It looks like you were submitting tasks manually.

[ogrisel]

Oops, sorry, that's was the wrong example. Here is the example with the joblib backend:

https://github.com/ogrisel/docker-distributed/blob/master/examples/sklearn_parameter_search_joblib.ipynb

[mrocklin]

That's really cool. Did you notice a speedup? Its hard to compare the numbers in the notebook directly.

I'll go over the joblib DistributedBackend sometime today, test it a bit more extensively and submit a PR, probably sometime tomorrow (traveling a bit today.)