[MRG] Fix auto memmap gc failure #294

ogrisel · 2016-01-11T14:29:35Z

The automatic memmap feature of joblib can be mislead by the reuse of id() values for recently garbage collected numpy arrays as demonstrated in the non-regression test of this PR. This problem is actually quite frequent when generating large numpy arrays with a Python generator.

This PR fixes the issues by systematically hashing the arrays to find a robust unique identifier for the filenames of the temporary memmap'ed arrays.

I ran the benchmark script varying the parameters and could not find a configuration where hashing could cause a very large performance degradation w.r.t. the current master.

This fixes scikit-learn/scikit-learn#6063.

GaelVaroquaux · 2016-01-11T15:33:51Z

joblib/pool.py

+        if context_id is not None:
+            warnings.warn('context_id is deprecated and ignored in joblib'
+                          ' 0.9.4 and will be removed in 0.11',
+                          DeprecationWarning)


I thhink that we should consider making pool.py private to not have to worry about these things.

I agree but I would rather not make this code change in the bugfix release to keep the diff readable.

lesteve · 2016-01-12T12:57:38Z

BTW this AppVeyor failure seems genuine (from there):

======================================================================
ERROR: joblib.test.test_parallel.test_auto_memmap_on_arrays_from_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "C:\Python27\lib\site-packages\joblib\test\test_parallel.py", line 575, in test_auto_memmap_on_arrays_from_generator
    delayed(check_memmap)(a) for a in generate_arrays(100))
  File "C:\Python27\lib\site-packages\joblib\parallel.py", line 819, in __call__
    self._terminate_pool()
  File "C:\Python27\lib\site-packages\joblib\parallel.py", line 549, in _terminate_pool
    self._pool.terminate()  # terminate does a join()
  File "C:\Python27\lib\site-packages\joblib\pool.py", line 575, in terminate
    delete_folder(self._temp_folder)
  File "C:\Python27\lib\site-packages\joblib\pool.py", line 427, in delete_folder
    shutil.rmtree(folder_path)
  File "C:\Python27\lib\shutil.py", line 252, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\Python27\lib\shutil.py", line 250, in rmtree
    os.remove(fullname)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\appveyor\\appdata\\local\\temp\\1\\joblib_memmaping_pool_2860_48143952\\2860-81511728-03797be121e43221444fc2914fb93286.pkl_01.npy'

GaelVaroquaux · 2016-01-12T14:29:13Z

joblib/test/test_pool.py

@@ -406,7 +391,7 @@ def test_memmaping_on_dev_shm():
        # pickling procedure generate a .pkl and a .npy file:
        assert_equal(len(os.listdir(pool_temp_folder)), 2)

-        b = np.ones(100, dtype=np.float64)
+        b = np.ones(100, dtype=np.float64) * 2


Why the "* 2"?

otherwise the 2 arrays a and b have the same content and I cannot test that b has been memmaped to a new file.

otherwise the 2 arrays a and b have the same content and I cannot test that b has been memmaped to a new file.

Good point. You should add a comment about that. Are we memmapping with 'c' or 'r'? The reason that I am asking is that if we are not, we need to memmap two different arrays (is the sense of identity) with the same content to different files.

we are memmaping with r (because of a windows specific constraint). I will add a comment.

ogrisel · 2016-01-13T13:29:46Z

@lesteve comments addressed. The status is red because coverage has decreased because of the new windows specific code, we can ignore it.

lesteve · 2016-01-13T14:10:33Z

joblib/pool.py

+                raise
+            else:
+                i += 1
+                sleep(delay)


I think you would never raise because i < n_retries. Also a for loop may be a tad simpler.

for i in range(n_retries) try: if os.path.exists(folder_path): shutil.rmtree(folder_path) break except WindowsError: # A worker process might still hold an open file descriptors # on one of the memory mapped arrays in the temporary folder # let's wait a bit an retry if i >= n_retries - 1: raise else: sleep(delay)

lesteve · 2016-01-13T15:42:25Z

Hmmm now AppVeyor fails in case you have missed it ...

ogrisel · 2016-01-14T09:13:46Z

Yes I am looking into it.

ogrisel · 2016-01-18T08:01:08Z

joblib/test/test_parallel.py

+    if not isinstance(a, np.memmap):
+        raise TypeError('Expected np.memmap instance, got %r',
+                        type(a))
+    return a.copy()  # return a regular array instead of a memmap


This was the cause of the previous failure under windows: passing a temporary memmap instance back to the master process prevents the deletion of the pool temp folder as long as. I changed the delete_folder method to tolerate this (rare case) warning the user.

Under POSIX, there is no problem: the folder is deleted while the memmap instances are still fully functional until they get garbage collected.

ogrisel · 2016-01-18T10:46:32Z

Ok I added a changelog entry, merging this.

[MRG] Fix auto memmap gc failure

* tag '0.9.4': (46 commits) Release 0.9.4 DOC add missing changelog entry for joblib#296 DOC add entry to changelog for joblib#294 ENH spare one file descriptor / syscall in automemmap FIX auto-memmap gc bug by always hashing arrays TST non-regression test for auto-memmap / gc bug Add link to github issues for 0.9.4 changelog entries Fix my_exceptions._mk_exception when input exception is not inheritable add entry in changelog fixing hashing with mixed dtype + test Use _compat.PY3_OR_LATER where possible COSMIT fix some PEP8 horizontal misalignments Move definition of PY3_OR_LATER to _compat.py Do not use inspect.getargspec FIX joblib#295: deadlock between async dispatch and exception handling Add section in CHANGES.rst TRAVIS use numpy 1.10 FIX style and pyflakes in test_pool.py Fix Parallel hanging with exhausted iterator remove useless section about versions of python prior to 2.6 ...

ogrisel added the bug label Jan 11, 2016

ogrisel changed the title ~~Fix auto memmap gc failure~~ [MRG] Fix auto memmap gc failure Jan 11, 2016

GaelVaroquaux reviewed Jan 11, 2016
View reviewed changes

GaelVaroquaux reviewed Jan 12, 2016
View reviewed changes

ogrisel force-pushed the fix-auto-memmap-gc-failure branch from 41b8fa1 to 7dc1c8a Compare January 13, 2016 10:37

ogrisel added this to the 0.9.4 milestone Jan 13, 2016

ogrisel force-pushed the fix-auto-memmap-gc-failure branch from fc6dea7 to 82964c7 Compare January 13, 2016 13:34

lesteve reviewed Jan 13, 2016
View reviewed changes

ogrisel force-pushed the fix-auto-memmap-gc-failure branch from 82964c7 to 6109123 Compare January 13, 2016 14:59

ogrisel force-pushed the fix-auto-memmap-gc-failure branch from 6109123 to e51760f Compare January 14, 2016 15:28

ogrisel reviewed Jan 18, 2016
View reviewed changes

ogrisel added 4 commits January 18, 2016 09:08

TST non-regression test for auto-memmap / gc bug

e3d084d

FIX auto-memmap gc bug by always hashing arrays

a6a0d26

ENH spare one file descriptor / syscall in automemmap

70330b3

DOC add entry to changelog for joblib#294

552c514

ogrisel force-pushed the fix-auto-memmap-gc-failure branch from e51760f to 552c514 Compare January 18, 2016 10:29

ogrisel added a commit that referenced this pull request Jan 18, 2016

Merge pull request #294 from ogrisel/fix-auto-memmap-gc-failure

b6af20c

[MRG] Fix auto memmap gc failure

ogrisel merged commit b6af20c into joblib:master Jan 18, 2016

GaelVaroquaux mentioned this pull request Jan 18, 2016

KFold on large data yields overlapping train and test sets scikit-learn/scikit-learn#6063

Closed

pyup-bot mentioned this pull request Jul 28, 2017

Pin joblib to latest version 0.11 workforce-data-initiative/skills-airflow#38

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MRG] Fix auto memmap gc failure #294

[MRG] Fix auto memmap gc failure #294

ogrisel commented Jan 11, 2016

GaelVaroquaux Jan 11, 2016

ogrisel Jan 13, 2016

lesteve commented Jan 12, 2016

GaelVaroquaux Jan 12, 2016

ogrisel Jan 13, 2016

GaelVaroquaux Jan 13, 2016 via email

ogrisel Jan 13, 2016

ogrisel commented Jan 13, 2016

lesteve Jan 13, 2016

ogrisel Jan 13, 2016

lesteve commented Jan 13, 2016

ogrisel commented Jan 14, 2016

ogrisel Jan 18, 2016

ogrisel commented Jan 18, 2016

[MRG] Fix auto memmap gc failure #294

[MRG] Fix auto memmap gc failure #294

Conversation

ogrisel commented Jan 11, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lesteve commented Jan 12, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GaelVaroquaux Jan 13, 2016 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ogrisel commented Jan 13, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lesteve commented Jan 13, 2016

ogrisel commented Jan 14, 2016

Choose a reason for hiding this comment

ogrisel commented Jan 18, 2016