New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fcma parallelize cv #176

Merged
merged 5 commits into from Feb 7, 2017

Conversation

Projects
None yet
2 participants
@yidawang
Contributor

yidawang commented Feb 3, 2017

Use pathos.multiprocessing to parallelize the cross-validation step of correlation-based voxel selection when sklearn.svm.SVC with precomputed kernel is used as the classifier.

Thanks in advance for reviewing!

@@ -28,6 +28,7 @@
from . import fcma_extension
from . import cython_blas as blas
import logging
import pathos.multiprocessing

This comment has been minimized.

@mihaic

mihaic Feb 6, 2017

Contributor

Given that you do not use other features of Pathos, could you please use the standalone Multiprocess package?
https://pypi.python.org/pypi/multiprocess
We are trying to remove the dependency on Pathos (see issue #149).

@mihaic

mihaic Feb 6, 2017

Contributor

Given that you do not use other features of Pathos, could you please use the standalone Multiprocess package?
https://pypi.python.org/pypi/multiprocess
We are trying to remove the dependency on Pathos (see issue #149).

This comment has been minimized.

@yidawang

yidawang Feb 6, 2017

Contributor

According to @mjanderson09 , the standalone multiprocessing can only serialize methods at the root level, and the Pathos version is more flexible so that we can define a lambda function and use it in the map method.

@yidawang

yidawang Feb 6, 2017

Contributor

According to @mjanderson09 , the standalone multiprocessing can only serialize methods at the root level, and the Pathos version is more flexible so that we can define a lambda function and use it in the map method.

This comment has been minimized.

@mihaic

mihaic Feb 6, 2017

Contributor

Multiprocess can serialize lambdas, as shown in the last example in its README:
https://github.com/uqfoundation/multiprocess#examples

@mihaic

mihaic Feb 6, 2017

Contributor

Multiprocess can serialize lambdas, as shown in the last example in its README:
https://github.com/uqfoundation/multiprocess#examples

This comment has been minimized.

@yidawang

yidawang Feb 6, 2017

Contributor

Sorry I didn't make myself clear. It is not just about lambda functions. A lambda function without involving any other user-defined methods should probably be fine, but in my case, I have _cross_validation_for_one_voxel used in my lambda function. If I want to use the standalone multiprocessing, I will have to put _cross_validation_for_one_voxel outside the class and define it at the root level.

@yidawang

yidawang Feb 6, 2017

Contributor

Sorry I didn't make myself clear. It is not just about lambda functions. A lambda function without involving any other user-defined methods should probably be fine, but in my case, I have _cross_validation_for_one_voxel used in my lambda function. If I want to use the standalone multiprocessing, I will have to put _cross_validation_for_one_voxel outside the class and define it at the root level.

@mihaic

mihaic approved these changes Feb 7, 2017

@mihaic mihaic merged commit eb02db4 into brainiak:master Feb 7, 2017

3 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
linux Build finished.
Details
macos Build finished.
Details

@yidawang yidawang deleted the yidawang:fcma_parallelize_cv branch Feb 7, 2017

danielsuo pushed a commit that referenced this pull request Nov 16, 2017

danielsuo pushed a commit that referenced this pull request Nov 16, 2017

New web UI. (#176)
* remove node.js webui

* temp commit

* flesh out web ui

* add documentation

* add ray timeline

* Small changes to documentation and formatting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment