Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use the resource_tracker to unlink shared temporary memmaps #966

Merged
merged 81 commits into from
May 3, 2020
Merged

Use the resource_tracker to unlink shared temporary memmaps #966

merged 81 commits into from
May 3, 2020

Commits on Mar 30, 2020

  1. try to remove preparation code in failing example

    @pierreglaser
    pierreglaser committed Mar 30, 2020
    Configuration menu
    Copy the full SHA
    e91e716 View commit details
    Browse the repository at this point in the history

  2. remove pickling workarounds related to old sphinx-gallery

    @pierreglaser
    pierreglaser committed Mar 30, 2020
    Configuration menu
    Copy the full SHA
    431d4c3 View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2020

  1. Merge remote-tracking branch 'upstream/master'

    @pierreglaser
    pierreglaser committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    550d66e View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2020

  1. Merge branch 'master' of github.com:joblib/joblib

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    d83b6d1 View commit details
    Browse the repository at this point in the history

  2. vendor loky 2.6.1

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    7366062 View commit details
    Browse the repository at this point in the history

  3. add the new file_plus_plus resource type

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    10ffeca View commit details
    Browse the repository at this point in the history

  4. TST add a new test reproducing the crash in joblib#806

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    c13a157 View commit details
    Browse the repository at this point in the history

  5. FIX delegate memmap cleanup to resource_tracker

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    4a077b8 View commit details
    Browse the repository at this point in the history

  6. TST WIP add initial reproducer of joblib#806 as a test

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    832438f View commit details
    Browse the repository at this point in the history

  7. MNT port vendor_loky.sh over to osx and windows

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    ee67f94 View commit details
    Browse the repository at this point in the history

  8. FIX mature refcounting of memmaps

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    f6d75bb View commit details
    Browse the repository at this point in the history

  9. ENH start re-writting delete_folder

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    363c69b View commit details
    Browse the repository at this point in the history

  10. temporary changes of the tests

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    4614a98 View commit details
    Browse the repository at this point in the history

  11. limit race possibility in file_plus_plus_unlink

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    7461081 View commit details
    Browse the repository at this point in the history

  12. log using mp.util.debug instead of print

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    85b92e3 View commit details
    Browse the repository at this point in the history

  13. start tracking temporary memmaps

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    abf7336 View commit details
    Browse the repository at this point in the history

  14. test cleanup + adapt current tests to new changes

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    de43355 View commit details
    Browse the repository at this point in the history

  15. python2/3 compat

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    cd7f274 View commit details
    Browse the repository at this point in the history

  16. use weakref.ref instead of finalize (python2 compatible)

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    6f8b1be View commit details
    Browse the repository at this point in the history

  17. unused imports

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    87aeedf View commit details
    Browse the repository at this point in the history

  18. fix loky vendoring issues

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    73ba88b View commit details
    Browse the repository at this point in the history

  19. unused imports

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    127bb2a View commit details
    Browse the repository at this point in the history

  20. run new tests for all multiprocessing backends

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    f67676e View commit details
    Browse the repository at this point in the history

  21. give time to a linux-only test to cleanup

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    f0de5fe View commit details
    Browse the repository at this point in the history

  22. fix assertion in prev test

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    b46a0da View commit details
    Browse the repository at this point in the history

  23. weakref python2/3 compat + increase rtracker robustness

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    92e0837 View commit details
    Browse the repository at this point in the history

  24. unused imports

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    9d3ce9e View commit details
    Browse the repository at this point in the history

  25. Include reproducer of joblib#942

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    7bb5f8f View commit details
    Browse the repository at this point in the history

  26. disable memmaping for multiprocessing backend

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    7cd1113 View commit details
    Browse the repository at this point in the history

  27. treat memmap as files, restore delete_folder logic

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    19f33fb View commit details
    Browse the repository at this point in the history

  28. add an allow_non_empty option to delete_folder

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    47ae7f9 View commit details
    Browse the repository at this point in the history

  29. serialize temporary memmaps as np.array in childs

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    edcbe83 View commit details
    Browse the repository at this point in the history

  30. make flake8 happy

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    5e1313c View commit details
    Browse the repository at this point in the history

  31. unused imports

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    bdb9388 View commit details
    Browse the repository at this point in the history

  32. restore memory mapping in multiprocessing backends

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    575a9cf View commit details
    Browse the repository at this point in the history

  33. linting

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    6ed5068 View commit details
    Browse the repository at this point in the history

  34. improve the reproducer (spawn-compatibility)

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    4b4be4e View commit details
    Browse the repository at this point in the history

  35. add new testutils file

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    18cdfbd View commit details
    Browse the repository at this point in the history

  36. some more if..main guards in tests

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    082754a View commit details
    Browse the repository at this point in the history

  37. limit potential race condition in a test

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    01dbb94 View commit details
    Browse the repository at this point in the history

  38. fix a bug making children still sending memmaps

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    e4339e9 View commit details
    Browse the repository at this point in the history

  39. remove duplicated function

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    6cf0deb View commit details
    Browse the repository at this point in the history

  40. unused imports

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    9774a40 View commit details
    Browse the repository at this point in the history

  41. catch for OSError when delerting folder at exit

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    2b41215 View commit details
    Browse the repository at this point in the history

  42. refactor test and add some more

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    cc1c95d View commit details
    Browse the repository at this point in the history

  43. various fixes/cleanups/test enhancements

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    c02f1f2 View commit details
    Browse the repository at this point in the history

  44. Apply suggestions from code review 

    Co-Authored-By: Olivier Grisel <olivier.grisel@ensta.org>
    @pierreglaser @ogrisel
    pierreglaser and ogrisel committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    3364c25 View commit details
    Browse the repository at this point in the history

  45. Update joblib/test/test_memmapping.py

    @ogrisel @pierreglaser
    ogrisel authored and pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    a7613a9 View commit details
    Browse the repository at this point in the history

  46. separate backward/forward reducers logic for ndarrays 

    As a side effect, JOBLIB_MEMMAPS is now only use is child processes.
    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    5de7b4d View commit details
    Browse the repository at this point in the history

  47. remove unused imports

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    e8a9ea5 View commit details
    Browse the repository at this point in the history

  48. remove stale usage of JOBLIB_MEMMAPS

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    a68d0cf View commit details
    Browse the repository at this point in the history

  49. fixup! remove stale usage of JOBLIB_MEMMAPS

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    f56e16f View commit details
    Browse the repository at this point in the history

  50. put resource management logic in a separate mixin

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    a69a7aa View commit details
    Browse the repository at this point in the history

  51. mock TemporaryResourcesManager when no multiprocessing

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    b5d35da View commit details
    Browse the repository at this point in the history

  52. track_memmap_in_child -> unlink_on_gc_collect

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    c6f222f View commit details
    Browse the repository at this point in the history

  53. remove reduce_memmap (unused function)

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    9d91108 View commit details
    Browse the repository at this point in the history

  54. reformulate a comment

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    18f264a View commit details
    Browse the repository at this point in the history

  55. don't pass kwargs as args in shutil.rmtree cal

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    f85eafd View commit details
    Browse the repository at this point in the history

  56. echo an informative error message in delete_folder

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    a670ece View commit details
    Browse the repository at this point in the history

  57. correct a few typos in a nearby comment

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    7a52c64 View commit details
    Browse the repository at this point in the history

  58. mention unlink_on_gc_collect in make_memmap doc

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    b68c573 View commit details
    Browse the repository at this point in the history

  59. rename crash->print_filename_and_raise

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    3bbee3d View commit details
    Browse the repository at this point in the history

  60. document what test_child...no_resource_leak tests

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    0f4220e View commit details
    Browse the repository at this point in the history

  61. test_permission_error: make sure stdout is empty

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    a7f4965 View commit details
    Browse the repository at this point in the history

  62. rename testutils.test_data -> return_slice_of_data

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    e2e3b1e View commit details
    Browse the repository at this point in the history

  63. rephrase comment in reduce_array_memmap_backward

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    bc0cd49 View commit details
    Browse the repository at this point in the history

  64. remove vendor_loky.ps1 (to be discussed)

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    38256dd View commit details
    Browse the repository at this point in the history

  65. remove Python 2 weakref compat trick

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    e161ba5 View commit details
    Browse the repository at this point in the history

  66. unused import

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    c4e75d6 View commit details
    Browse the repository at this point in the history

  67. add some pytest pragmas

    @pierreglaser
    pierreglaser committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    2c8e9ac View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2020

  1. add allow_empty arg to try_delete_folder

    @pierreglaser
    pierreglaser committed Apr 23, 2020
    Configuration menu
    Copy the full SHA
    152c9d8 View commit details
    Browse the repository at this point in the history

  2. maximum explicitness when calling delete_folder

    @pierreglaser
    pierreglaser committed Apr 23, 2020
    Configuration menu
    Copy the full SHA
    120d2ad View commit details
    Browse the repository at this point in the history

  3. Parallel fails: delete temp files after __call__ 

    Previously, the temporary folder temporary was cleaned during
the atexit finalizer
    @pierreglaser
    pierreglaser committed Apr 23, 2020
    Configuration menu
    Copy the full SHA
    62669d3 View commit details
    Browse the repository at this point in the history

  4. add a missing with_multiprocessing pragma

    @pierreglaser
    pierreglaser committed Apr 23, 2020
    Configuration menu
    Copy the full SHA
    8303c92 View commit details
    Browse the repository at this point in the history

  5. minor touch-ups 

    Co-Authored-By: Olivier Grisel <olivier.grisel@gmail.com>
    @pierreglaser @ogrisel
    pierreglaser and ogrisel committed Apr 23, 2020
    Configuration menu
    Copy the full SHA
    24cf682 View commit details
    Browse the repository at this point in the history

  6. test this PR against windows + Python3.6

    @pierreglaser
    pierreglaser committed Apr 23, 2020
    Configuration menu
    Copy the full SHA
    0f3c229 View commit details
    Browse the repository at this point in the history

  7. test against Python 3.7 + windows

    @pierreglaser
    pierreglaser committed Apr 23, 2020
    Configuration menu
    Copy the full SHA
    0e967c0 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2020

  1. retry sensitive tests many times in test suite

    @pierreglaser
    pierreglaser committed Apr 24, 2020
    Configuration menu
    Copy the full SHA
    31f74f3 View commit details
    Browse the repository at this point in the history

  2. repeat the sensitive test 50 (!) times

    @pierreglaser
    pierreglaser committed Apr 24, 2020
    Configuration menu
    Copy the full SHA
    39428f6 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2020

  1. remove debug code

    @pierreglaser
    pierreglaser committed Apr 27, 2020
    Configuration menu
    Copy the full SHA
    375897c View commit details
    Browse the repository at this point in the history

Commits on May 3, 2020

  1. Merge branch 'master' into memmap_reference_counting

    @ogrisel
    ogrisel committed May 3, 2020
    Configuration menu
    Copy the full SHA
    ce67429 View commit details
    Browse the repository at this point in the history