New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fcma parallelize cv #176
Fcma parallelize cv #176
Conversation
…n correlation-based voxel selection
…ta when using multiprocessing
@@ -28,6 +28,7 @@ | |||
from . import fcma_extension | |||
from . import cython_blas as blas | |||
import logging | |||
import pathos.multiprocessing |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given that you do not use other features of Pathos, could you please use the standalone Multiprocess package?
https://pypi.python.org/pypi/multiprocess
We are trying to remove the dependency on Pathos (see issue #149).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
According to @mjanderson09 , the standalone multiprocessing can only serialize methods at the root level, and the Pathos version is more flexible so that we can define a lambda function and use it in the map
method.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Multiprocess can serialize lambdas, as shown in the last example in its README:
https://github.com/uqfoundation/multiprocess#examples
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry I didn't make myself clear. It is not just about lambda functions. A lambda function without involving any other user-defined methods should probably be fine, but in my case, I have _cross_validation_for_one_voxel
used in my lambda function. If I want to use the standalone multiprocessing, I will have to put _cross_validation_for_one_voxel
outside the class and define it at the root level.
* remove node.js webui * temp commit * flesh out web ui * add documentation * add ray timeline * Small changes to documentation and formatting.
Use
pathos.multiprocessing
to parallelize the cross-validation step of correlation-based voxel selection whensklearn.svm.SVC
with precomputed kernel is used as the classifier.Thanks in advance for reviewing!