Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fcma parallelize cv #176

Merged
merged 5 commits into from Feb 7, 2017
Merged

Fcma parallelize cv #176

merged 5 commits into from Feb 7, 2017

Conversation

@yidawang
Copy link
Member

@yidawang yidawang commented Feb 3, 2017

Use pathos.multiprocessing to parallelize the cross-validation step of correlation-based voxel selection when sklearn.svm.SVC with precomputed kernel is used as the classifier.

Thanks in advance for reviewing!

@@ -28,6 +28,7 @@
from . import fcma_extension
from . import cython_blas as blas
import logging
import pathos.multiprocessing

This comment has been minimized.

@mihaic

mihaic Feb 6, 2017
Contributor

Given that you do not use other features of Pathos, could you please use the standalone Multiprocess package?
https://pypi.python.org/pypi/multiprocess
We are trying to remove the dependency on Pathos (see issue #149).

This comment has been minimized.

@yidawang

yidawang Feb 6, 2017
Author Member

According to @mjanderson09 , the standalone multiprocessing can only serialize methods at the root level, and the Pathos version is more flexible so that we can define a lambda function and use it in the map method.

This comment has been minimized.

@mihaic

mihaic Feb 6, 2017
Contributor

Multiprocess can serialize lambdas, as shown in the last example in its README:
https://github.com/uqfoundation/multiprocess#examples

This comment has been minimized.

@yidawang

yidawang Feb 6, 2017
Author Member

Sorry I didn't make myself clear. It is not just about lambda functions. A lambda function without involving any other user-defined methods should probably be fine, but in my case, I have _cross_validation_for_one_voxel used in my lambda function. If I want to use the standalone multiprocessing, I will have to put _cross_validation_for_one_voxel outside the class and define it at the root level.

@mihaic
mihaic approved these changes Feb 7, 2017
@mihaic mihaic merged commit eb02db4 into brainiak:master Feb 7, 2017
3 checks passed
3 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
linux Build finished.
Details
macos Build finished.
Details
@yidawang yidawang deleted the yidawang:fcma_parallelize_cv branch Feb 7, 2017
danielsuo pushed a commit that referenced this pull request Nov 16, 2017
danielsuo pushed a commit that referenced this pull request Nov 16, 2017
* remove node.js webui

* temp commit

* flesh out web ui

* add documentation

* add ray timeline

* Small changes to documentation and formatting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants