Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+1] Remove warnings in nested Parallel when the inner Parallel has n_jobs=1 #406

Merged
merged 8 commits into from Oct 5, 2016

Conversation

@harishmk
Copy link
Contributor

harishmk commented Sep 30, 2016

suppress a warning if a daemonic process creates parallel jobs with n_jobs=1
#405

suppress a warning if a daemonic process creates parallel jobs with n_jobs=1
@GaelVaroquaux GaelVaroquaux changed the title Update _parallel_backends.py [MRG+1] Update _parallel_backends.py Oct 1, 2016
@GaelVaroquaux

This comment has been minimized.

Copy link
Member

GaelVaroquaux commented Oct 1, 2016

LGTM. +1 for merge. Thank you!

@aabadie

This comment has been minimized.

Copy link
Contributor

aabadie commented Oct 1, 2016

I could reproduce the initial problem mentionned in #405 on Windows with python 2.7. The fix looks good to me.
One thing is missing though : a test. Can you add a test function in joblib/test/test_parallel.py that checks no warning is raised when n_jobs = 1 ?

@lesteve lesteve added this to the 0.10.3 milestone Oct 5, 2016
@lesteve

This comment has been minimized.

Copy link
Member

lesteve commented Oct 5, 2016

A snippet reproducing the problem:

from joblib import Parallel, delayed


def func():
    return 42


def inner_parallel_func():
    Parallel(n_jobs=1)(delayed(func)() for _ in range(10))

if __name__ == '__main__':
    Parallel(n_jobs=2)(delayed(inner_parallel_func)() for _ in range(10))
@lesteve lesteve changed the title [MRG+1] Update _parallel_backends.py [MRG+1] Remove warnings in nested Parallel when the inner Parallel has n_jobs=1 Oct 5, 2016
Enhance check_subprocess_call to be able to check stderr as well as
stdout.
@lesteve

This comment has been minimized.

Copy link
Member

lesteve commented Oct 5, 2016

I used this as an excuse to test the recent features of github where PR authors can allow the maintainers to push commits on the PR branch and it seems to work just fine, this is very nifty!

I added a test for the warnings, as you can see from the diff it's not that easy to catch the warnings because they happen in child processes so I reuse and enhanced check_subprocess_call ...

@lesteve lesteve force-pushed the harishmk:patch-1 branch to 1a2dc68 Oct 5, 2016
@lesteve

This comment has been minimized.

Copy link
Member

lesteve commented Oct 5, 2016

@aabadie do you want to have a look at this one before I merge?

def parallel_func():
res = Parallel(n_jobs={inner_n_jobs})(delayed(func)() for _ in range(3))
# Needed otherwise some warnings may be missed

This comment has been minimized.

Copy link
@aabadie

aabadie Oct 5, 2016

Contributor

not super critical but I think this comment is useless or there's something missing after.

This comment has been minimized.

Copy link
@lesteve

lesteve Oct 5, 2016

Member

good point, I tried different ways of flushing stderr to get the number of warnings I expected to no avail so I eventually gave up, except I forgot the comment there.

@aabadie
aabadie approved these changes Oct 5, 2016
Copy link
Contributor

aabadie left a comment

I admit adding a test for this was a bit tricky :)

@lesteve

This comment has been minimized.

Copy link
Member

lesteve commented Oct 5, 2016

I admit adding a test for this was a bit tricky :)

I was quite happy that check_subprocess_call (originally here for checking that a function defined in __main__ can be used inside Parallel, which is very useful in interactive i.e. IPython settings) could be reused without too much pain for the test.

@aabadie

This comment has been minimized.

Copy link
Contributor

aabadie commented Oct 5, 2016

Merging this one. Thanks @harishmk and @lesteve

@aabadie aabadie merged commit 5f169bb into joblib:master Oct 5, 2016
3 checks passed
3 checks passed
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage decreased (-0.05%) to 89.602%
Details
lesteve added a commit that referenced this pull request Oct 6, 2016
…s n_jobs=1 (#406)

* Update _parallel_backends.py

suppress a warning if a daemonic process creates parallel jobs with n_jobs=1

* Add test for warnings in nested Parallel calls

Enhance check_subprocess_call to be able to check stderr as well as
stdout.
@lesteve

This comment has been minimized.

Copy link
Member

lesteve commented Oct 18, 2016

@harishmk do you mind giving me your full name ?

I would need it for CHANGES.rst since I am planning to do a bug-fix release of joblib (mainly for scikit-learn 0.18.1) and this PR would be part of it. If I don't hear from you, I'll just use your github user name.

yarikoptic added a commit to yarikoptic/joblib that referenced this pull request Dec 8, 2016
* tag '0.10.3':
  MAINT simplify setup.py
  Bump up version for 0.10.3 release
  Add entries for 0.10.3 release in CHANGES.rst
  [MRG+1] Fix tests when multiprocessing is disabled (joblib#409)
  [MRG+1] Remove warnings in nested Parallel when the inner Parallel has n_jobs=1 (joblib#406)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.