Cached function executed in parallel yield corrupted cache and unnecessary recomputations

[lesteve]

From scikit-learn/scikit-learn#7990 (comment).

Snippet reproducing the problem:

from joblib import Parallel, delayed, Memory

mem = Memory('/tmp/joblib', verbose=10)


def func():
    complex_obj = ['a'*1000] * 1000
    return complex_obj


func_cached = mem.cache(func)

Parallel(n_jobs=-1)(delayed(func_cached)() for i in range(100))

Run with:

rm -rf /tmp/joblib/ && ipython /tmp/test.py 2>&1 | tee log

Plenty of warnings like this:

WARNING:root:[MemorizedFunc(func=<function func at 0x7f41f372d048>, cachedir='/tmp/joblib/joblib')]: Exception
 Traceback (most recent call last):
  File "/home/lesteve/dev/joblib/joblib/memory.py", line 514, in _cached_call
    verbose=self._verbose)
  File "/home/lesteve/dev/joblib/joblib/memory.py", line 138, in _load_output
    result = numpy_pickle.load(filename, mmap_mode=mmap_mode)
  File "/home/lesteve/dev/joblib/joblib/numpy_pickle.py", line 578, in load
    obj = _unpickle(fobj, filename, mmap_mode)
  File "/home/lesteve/dev/joblib/joblib/numpy_pickle.py", line 508, in _unpickle
    obj = unpickler.load()
  File "/home/lesteve/miniconda3/lib/python3.5/pickle.py", line 1037, in load
    raise EOFError
EOFError