[MRG+1] Remove warnings in nested Parallel when the inner Parallel has n_jobs=1 - #406
Conversation
suppress a warning if a daemonic process creates parallel jobs with n_jobs=1
GaelVaroquaux
changed the title
|
LGTM. +1 for merge. Thank you! |
|
I could reproduce the initial problem mentionned in #405 on Windows with python 2.7. The fix looks good to me. |
|
A snippet reproducing the problem: from joblib import Parallel, delayed
def func():
return 42
def inner_parallel_func():
Parallel(n_jobs=1)(delayed(func)() for _ in range(10))
if __name__ == '__main__':
Parallel(n_jobs=2)(delayed(inner_parallel_func)() for _ in range(10)) |
lesteve
changed the title
Enhance check_subprocess_call to be able to check stderr as well as stdout.
|
I used this as an excuse to test the recent features of github where PR authors can allow the maintainers to push commits on the PR branch and it seems to work just fine, this is very nifty! I added a test for the warnings, as you can see from the diff it's not that easy to catch the warnings because they happen in child processes so I reuse and enhanced check_subprocess_call ... |
|
@aabadie do you want to have a look at this one before I merge? |
|
|
||
| def parallel_func(): | ||
| res = Parallel(n_jobs={inner_n_jobs})(delayed(func)() for _ in range(3)) | ||
| # Needed otherwise some warnings may be missed |
There was a problem hiding this comment.
not super critical but I think this comment is useless or there's something missing after.
There was a problem hiding this comment.
good point, I tried different ways of flushing stderr to get the number of warnings I expected to no avail so I eventually gave up, except I forgot the comment there.
aabadie
left a comment
aabadie
left a comment
There was a problem hiding this comment.
I admit adding a test for this was a bit tricky :)
I was quite happy that check_subprocess_call (originally here for checking that a function defined in |
…s n_jobs=1 (#406) * Update _parallel_backends.py suppress a warning if a daemonic process creates parallel jobs with n_jobs=1 * Add test for warnings in nested Parallel calls Enhance check_subprocess_call to be able to check stderr as well as stdout.
|
@harishmk do you mind giving me your full name ? I would need it for CHANGES.rst since I am planning to do a bug-fix release of joblib (mainly for scikit-learn 0.18.1) and this PR would be part of it. If I don't hear from you, I'll just use your github user name. |
* tag '0.10.3': MAINT simplify setup.py Bump up version for 0.10.3 release Add entries for 0.10.3 release in CHANGES.rst [MRG+1] Fix tests when multiprocessing is disabled (joblib#409) [MRG+1] Remove warnings in nested Parallel when the inner Parallel has n_jobs=1 (joblib#406)
4 participants
suppress a warning if a daemonic process creates parallel jobs with n_jobs=1
#405