[MRG] Fix auto memmap gc failure

[ogrisel]

The automatic memmap feature of joblib can be mislead by the reuse of id() values for recently garbage collected numpy arrays as demonstrated in the non-regression test of this PR. This problem is actually quite frequent when generating large numpy arrays with a Python generator.

This PR fixes the issues by systematically hashing the arrays to find a robust unique identifier for the filenames of the temporary memmap'ed arrays.

I ran the benchmark script varying the parameters and could not find a configuration where hashing could cause a very large performance degradation w.r.t. the current master.

This fixes scikit-learn/scikit-learn#6063.

[GaelVaroquaux]

I thhink that we should consider making pool.py private to not have to worry about these things.

[ogrisel]

I agree but I would rather not make this code change in the bugfix release to keep the diff readable.

[lesteve]

BTW this AppVeyor failure seems genuine (from there):

======================================================================
ERROR: joblib.test.test_parallel.test_auto_memmap_on_arrays_from_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "C:\Python27\lib\site-packages\joblib\test\test_parallel.py", line 575, in test_auto_memmap_on_arrays_from_generator
    delayed(check_memmap)(a) for a in generate_arrays(100))
  File "C:\Python27\lib\site-packages\joblib\parallel.py", line 819, in __call__
    self._terminate_pool()
  File "C:\Python27\lib\site-packages\joblib\parallel.py", line 549, in _terminate_pool
    self._pool.terminate()  # terminate does a join()
  File "C:\Python27\lib\site-packages\joblib\pool.py", line 575, in terminate
    delete_folder(self._temp_folder)
  File "C:\Python27\lib\site-packages\joblib\pool.py", line 427, in delete_folder
    shutil.rmtree(folder_path)
  File "C:\Python27\lib\shutil.py", line 252, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\Python27\lib\shutil.py", line 250, in rmtree
    os.remove(fullname)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\appveyor\\appdata\\local\\temp\\1\\joblib_memmaping_pool_2860_48143952\\2860-81511728-03797be121e43221444fc2914fb93286.pkl_01.npy'

[GaelVaroquaux]

Why the "* 2"?

[ogrisel]

otherwise the 2 arrays a and b have the same content and I cannot test that b has been memmaped to a new file.

[GaelVaroquaux]

otherwise the 2 arrays a and b have the same content and I cannot test that b has been memmaped to a new file.

Good point. You should add a comment about that. Are we memmapping with 'c' or 'r'? The reason that I am asking is that if we are not, we need to memmap two different arrays (is the sense of identity) with the same content to different files.

[ogrisel]

we are memmaping with r (because of a windows specific constraint). I will add a comment.

[ogrisel]

@lesteve comments addressed. The status is red because coverage has decreased because of the new windows specific code, we can ignore it.

[lesteve]

I think you would never raise because i < n_retries. Also a for loop may be a tad simpler.

for i in range(n_retries)
    try:
        if os.path.exists(folder_path):
            shutil.rmtree(folder_path)
        break
    except WindowsError:
        # A worker process might still hold an open file descriptors
        # on one of the memory mapped arrays in the temporary folder
        # let's wait a bit an retry
        if i >= n_retries - 1:
            raise
        else:
            sleep(delay)

[ogrisel]

good catch

[lesteve]

Hmmm now AppVeyor fails in case you have missed it ...

[ogrisel]

Yes I am looking into it.

[ogrisel]

This was the cause of the previous failure under windows: passing a temporary memmap instance back to the master process prevents the deletion of the pool temp folder as long as. I changed the delete_folder method to tolerate this (rare case) warning the user.

Under POSIX, there is no problem: the folder is deleted while the memmap instances are still fully functional until they get garbage collected.

[ogrisel]

Ok I added a changelog entry, merging this.