Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed unnecessary Pool creation in searchlight #386

Merged
merged 3 commits into from Oct 16, 2018

Conversation

@davidt0x
Copy link
Contributor

commented Oct 15, 2018

When pool_size=1 the creation of a multiprocessing pool is unnecessary
and wasteful because data needs to be copied and sent to other process.
This would double the needed memory for each MPI task. In addition,
fork() can cause unpredictable behaviour in some MPI implementations,
see:

https://www.open-mpi.org/faq/?category=tuning#fork-warning

David Turner
Fixed unnecessary Pool creation in searchlight
When pool_size=1 the creation of a multiprocessing pool is unnecessary
and wasteful because data needs to be copied and sent to other process.
This would double the needed memory for each MPI task. In addition,
fork() can cause unpredictable behaviour in some MPI implementations,
see:

https://www.open-mpi.org/faq/?category=tuning#fork-warning
@buildbot-princeton

This comment has been minimized.

Copy link
Collaborator

commented Oct 15, 2018

Can one of the admins verify this patch?

1 similar comment
@buildbot-princeton

This comment has been minimized.

Copy link
Collaborator

commented Oct 15, 2018

Can one of the admins verify this patch?

@mihaic

This comment has been minimized.

Copy link
Contributor

commented Oct 15, 2018

Jenkins, add to whitelist.

@mihaic
Copy link
Contributor

left a comment

@davidt0x, thanks for the PR. It makes sense. Could you please fix the lint errors?

David Turner
@mihaic

This comment has been minimized.

Copy link
Contributor

commented Oct 16, 2018

Jenkins, retest this please.

@davidt0x

This comment has been minimized.

Copy link
Contributor Author

commented Oct 16, 2018

Do we need a unit test for pool_size = 1 to get coverage up? Am I reading this correctly?

@mihaic

This comment has been minimized.

Copy link
Contributor

commented Oct 16, 2018

Precisely!

@davidt0x

This comment has been minimized.

Copy link
Contributor Author

commented Oct 16, 2018

Ok, I copied one of the previous tests with the only difference being pool_size=1. Test passes and coverage was good on my end. Third times the charm hopefully :)

@mihaic
mihaic approved these changes Oct 16, 2018

@mihaic mihaic merged commit 8994231 into brainiak:master Oct 16, 2018

5 checks passed

codecov/patch 100% of diff hit (target 91.39%)
Details
codecov/project 91.45% (+0.05%) compared to 40a8d15
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
linux Build finished.
Details
macos Build finished.
Details
@mihaic

This comment has been minimized.

Copy link
Contributor

commented Oct 16, 2018

Thanks a lot, @davidt0x! This should make some people happy. :)

@davidt0x davidt0x deleted the davidt0x:searchlight_poolsize_1_opt branch Oct 17, 2018

mihaic added a commit to mihaic/brainiak that referenced this pull request Jun 21, 2019
mihaic added a commit to mihaic/brainiak that referenced this pull request Jun 21, 2019
mihaic added a commit that referenced this pull request Aug 14, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.