Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH only protect against nested call in Parallel #934

Merged
merged 7 commits into from
Sep 10, 2019

Conversation

tomMoral
Copy link
Contributor

@tomMoral tomMoral commented Sep 9, 2019

Another solution to fix #865 without explicitly asking for it.
Here, the protection against nested call is implemented to only be used in nested Parallel calls.

@tomMoral tomMoral force-pushed the ENH_parallel_run_threads branch from 99037a4 to e492f81 Compare September 9, 2019 10:20
@codecov
Copy link

codecov bot commented Sep 9, 2019

Codecov Report

Merging #934 into master will decrease coverage by 21.89%.
The diff coverage is 54.34%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master    #934      +/-   ##
=========================================
- Coverage    95.4%   73.5%   -21.9%     
=========================================
  Files          45      45              
  Lines        6460    6492      +32     
=========================================
- Hits         6163    4772    -1391     
- Misses        297    1720    +1423
Impacted Files Coverage Δ
joblib/test/test_parallel.py 60.16% <27.27%> (-36.81%) ⬇️
joblib/_parallel_backends.py 52.17% <62.5%> (-42.23%) ⬇️
joblib/parallel.py 84.61% <87.5%> (-12.67%) ⬇️
joblib/test/test_dask.py 4.47% <0%> (-94.03%) ⬇️
joblib/_dask.py 24.29% <0%> (-71.19%) ⬇️
joblib/_memory_helpers.py 3.07% <0%> (-70.77%) ⬇️
joblib/backports.py 29.16% <0%> (-64.59%) ⬇️
joblib/pool.py 28.44% <0%> (-62.94%) ⬇️
joblib/executor.py 37.93% <0%> (-62.07%) ⬇️
joblib/test/test_memmapping.py 37.46% <0%> (-61.99%) ⬇️
... and 23 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fd585ce...2c13ef6. Read the comment docs.

@codecov
Copy link

codecov bot commented Sep 9, 2019

Codecov Report

Merging #934 into master will increase coverage by 0.02%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #934      +/-   ##
==========================================
+ Coverage    95.4%   95.42%   +0.02%     
==========================================
  Files          45       45              
  Lines        6460     6494      +34     
==========================================
+ Hits         6163     6197      +34     
  Misses        297      297
Impacted Files Coverage Δ
joblib/test/test_parallel.py 97.05% <100%> (+0.08%) ⬆️
joblib/_parallel_backends.py 94.86% <100%> (+0.46%) ⬆️
joblib/parallel.py 97.33% <100%> (+0.05%) ⬆️
joblib/test/test_module.py 94.44% <0%> (-5.56%) ⬇️
joblib/hashing.py 93.16% <0%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fd585ce...3c32ec3. Read the comment docs.

@tomMoral tomMoral requested a review from ogrisel September 9, 2019 13:28
Copy link
Contributor

@ogrisel ogrisel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you very much for the fix. Please add an entry to the changelog!

@tomMoral tomMoral merged commit eb7a949 into joblib:master Sep 10, 2019
@tomMoral tomMoral deleted the ENH_parallel_run_threads branch September 10, 2019 10:20
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Oct 31, 2019
Release 0.14.0
Improved the load balancing between workers to avoid stranglers caused by an excessively large batch size when the task duration is varying significantly (because of the combined use of joblib.Parallel and joblib.Memory with a partially warmed cache for instance). joblib/joblib#899
Add official support for Python 3.8: fixed protocol number in Hasher and updated tests.
Fix a deadlock when using the dask backend (when scattering large numpy arrays). joblib/joblib#914
Warn users that they should never use joblib.load with files from untrusted sources. Fix security related API change introduced in numpy 1.6.3 that would prevent using joblib with recent numpy versions. joblib/joblib#879
Upgrade to cloudpickle 1.1.1 that add supports for the upcoming Python 3.8 release among other things. joblib/joblib#878
Fix semaphore availability checker to avoid spawning resource trackers on module import. joblib/joblib#893
Fix the oversubscription protection to only protect against nested Parallel calls. This allows joblib to be run in background threads. joblib/joblib#934
Fix ValueError (negative dimensions) when pickling large numpy arrays on Windows. joblib/joblib#920
Upgrade to loky 2.6.0 that add supports for the setting environment variables in child before loading any module. joblib/joblib#940
Fix the oversubscription protection for native libraries using threadpools (OpenBLAS, MKL, Blis and OpenMP runtimes). The maximal number of threads is can now be set in children using the inner_max_num_threads in parallel_backend. It defaults to cpu_count() // n_jobs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

not possible to use joblib as background thread
2 participants