Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upFcma parallelize cv #176
Conversation
yidawang
added some commits
Feb 3, 2017
| @@ -28,6 +28,7 @@ | ||
| from . import fcma_extension | ||
| from . import cython_blas as blas | ||
| import logging | ||
| import pathos.multiprocessing |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
mihaic
Feb 6, 2017
Contributor
Given that you do not use other features of Pathos, could you please use the standalone Multiprocess package?
https://pypi.python.org/pypi/multiprocess
We are trying to remove the dependency on Pathos (see issue #149).
mihaic
Feb 6, 2017
Contributor
Given that you do not use other features of Pathos, could you please use the standalone Multiprocess package?
https://pypi.python.org/pypi/multiprocess
We are trying to remove the dependency on Pathos (see issue #149).
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
yidawang
Feb 6, 2017
Contributor
According to @mjanderson09 , the standalone multiprocessing can only serialize methods at the root level, and the Pathos version is more flexible so that we can define a lambda function and use it in the map method.
yidawang
Feb 6, 2017
Contributor
According to @mjanderson09 , the standalone multiprocessing can only serialize methods at the root level, and the Pathos version is more flexible so that we can define a lambda function and use it in the map method.
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
mihaic
Feb 6, 2017
Contributor
Multiprocess can serialize lambdas, as shown in the last example in its README:
https://github.com/uqfoundation/multiprocess#examples
mihaic
Feb 6, 2017
Contributor
Multiprocess can serialize lambdas, as shown in the last example in its README:
https://github.com/uqfoundation/multiprocess#examples
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
yidawang
Feb 6, 2017
Contributor
Sorry I didn't make myself clear. It is not just about lambda functions. A lambda function without involving any other user-defined methods should probably be fine, but in my case, I have _cross_validation_for_one_voxel used in my lambda function. If I want to use the standalone multiprocessing, I will have to put _cross_validation_for_one_voxel outside the class and define it at the root level.
yidawang
Feb 6, 2017
Contributor
Sorry I didn't make myself clear. It is not just about lambda functions. A lambda function without involving any other user-defined methods should probably be fine, but in my case, I have _cross_validation_for_one_voxel used in my lambda function. If I want to use the standalone multiprocessing, I will have to put _cross_validation_for_one_voxel outside the class and define it at the root level.
mihaic
merged commit eb02db4
into
brainiak:master
Feb 7, 2017
yidawang
deleted the
yidawang:fcma_parallelize_cv
branch
Feb 7, 2017
pushed a commit
that referenced
this pull request
Nov 16, 2017
pushed a commit
that referenced
this pull request
Nov 16, 2017
yidawang commentedFeb 3, 2017
Use
pathos.multiprocessingto parallelize the cross-validation step of correlation-based voxel selection whensklearn.svm.SVCwith precomputed kernel is used as the classifier.Thanks in advance for reviewing!