Fcma parallelize cv #176
Fcma parallelize cv #176
Conversation
…n correlation-based voxel selection
…ta when using multiprocessing
| @@ -28,6 +28,7 @@ | |||
| from . import fcma_extension | |||
| from . import cython_blas as blas | |||
| import logging | |||
| import pathos.multiprocessing | |||
mihaic
Feb 6, 2017
Contributor
Given that you do not use other features of Pathos, could you please use the standalone Multiprocess package?
https://pypi.python.org/pypi/multiprocess
We are trying to remove the dependency on Pathos (see issue #149).
Given that you do not use other features of Pathos, could you please use the standalone Multiprocess package?
https://pypi.python.org/pypi/multiprocess
We are trying to remove the dependency on Pathos (see issue #149).
yidawang
Feb 6, 2017
Author
Member
According to @mjanderson09 , the standalone multiprocessing can only serialize methods at the root level, and the Pathos version is more flexible so that we can define a lambda function and use it in the map method.
According to @mjanderson09 , the standalone multiprocessing can only serialize methods at the root level, and the Pathos version is more flexible so that we can define a lambda function and use it in the map method.
mihaic
Feb 6, 2017
Contributor
Multiprocess can serialize lambdas, as shown in the last example in its README:
https://github.com/uqfoundation/multiprocess#examples
Multiprocess can serialize lambdas, as shown in the last example in its README:
https://github.com/uqfoundation/multiprocess#examples
yidawang
Feb 6, 2017
Author
Member
Sorry I didn't make myself clear. It is not just about lambda functions. A lambda function without involving any other user-defined methods should probably be fine, but in my case, I have _cross_validation_for_one_voxel used in my lambda function. If I want to use the standalone multiprocessing, I will have to put _cross_validation_for_one_voxel outside the class and define it at the root level.
Sorry I didn't make myself clear. It is not just about lambda functions. A lambda function without involving any other user-defined methods should probably be fine, but in my case, I have _cross_validation_for_one_voxel used in my lambda function. If I want to use the standalone multiprocessing, I will have to put _cross_validation_for_one_voxel outside the class and define it at the root level.
* remove node.js webui * temp commit * flesh out web ui * add documentation * add ray timeline * Small changes to documentation and formatting.
Use
pathos.multiprocessingto parallelize the cross-validation step of correlation-based voxel selection whensklearn.svm.SVCwith precomputed kernel is used as the classifier.Thanks in advance for reviewing!