Joblib.Parallel hangs when passed an empty iterable (python 2.7)

[joshloyal]

Currently the parallel loop will hang for the following code snippets

from math import sqrt
from joblib import Parallel, delayed

Parallel(n_jobs=2)(delayed(sqrt)(i) for i in range(0))

or even just

Parallel(n_jobs=2)([])

I'm not too familiar with the internals of the parallel loop, but it seems to be because we are requesting a parallel loop with an empty iterable (it works fine for n_jobs=1, i.e. a flat loop). Up until joblib==0.9.2 this would return an empty list.

[lesteve]

Can reproduce on both python 2 and python 3.

[aabadie]

I could make this work by simply adding self._iterating = False here: https://github.com/joblib/joblib/blob/master/joblib/parallel.py#L720

Otherwise the caller is stuck in an infinite loop.

[lesteve]

I could make this work by simply adding self._iterating = False here: https://github.com/joblib/joblib/blob/master/joblib/parallel.py#L720

Otherwise the caller is stuck in an infinite loop.

This doesn't feel like the right place to do it but I may be wrong. I double-checked that all the tests pass and I wasn't able to put together a snippet that showed a problem with your fix. To be continued ...

[aabadie]

This doesn't feel like the right place to do it but I may be wrong. I double-checked that all the tests pass and I wasn't able to put together a snippet that showed a problem with your fix. To be continued ...

I know but I couldn't find a better place for this. This code seems to be quite racy.

[lesteve]

I know but I couldn't find a better place for this. This code seems to be quite racy.

Racy as in subject to race condition I guess ;-). My personal attempt was to set self._iterating to False here and only set it to True if the body of the while loop just below is executed at least once here.

[aabadie]

Racy as in subject to race condition I guess ;-)

Exactly !

My personal attempt was to set self._iterating to False here and only set it to True if the body of the while loop just below is executed at least once here.

This makes sense and sounds like a better solution.

[aabadie]

@lesteve, I tried your solution, it works. I'm wondering if this could also solve the issue with appveyor.

[lesteve]

@lesteve, I tried your solution, it works. I'm wondering if this could also solve the issue with appveyor.

I'll put together a PR tomorrow for the empty iterator case and try to understand the code better with @ogrisel's help since he is the one who understand it best.

[lesteve]

I opened a PR #293. @ogrisel thinks the fix is fine.

[joshloyal]

Great thanks everyone!