Skip to content

Why do we use multiprocess instead of multiprocessing #60

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ericmjonas opened this issue Feb 23, 2017 · 5 comments · Fixed by #94
Closed

Why do we use multiprocess instead of multiprocessing #60

ericmjonas opened this issue Feb 23, 2017 · 5 comments · Fixed by #94
Milestone

Comments

@ericmjonas
Copy link
Collaborator

This seems like an unnecessary dependency given that we just 1. use it in the stand-alone mode and 2. I think multiprocessing provides all the same functionality? I'm still not sure what the multiprocess fork actually does.

@shivaram
Copy link
Collaborator

Just to clarify you mean what are the differences between multiprocess (https://github.com/uqfoundation/multiprocess) and multiprocessing (https://docs.python.org/2/library/multiprocessing.html) ?

@ericmjonas
Copy link
Collaborator Author

Yes, exactly -- I think I depended on multiprocess out of habit / it used to be the only game in town, but if we can get away with the (built-in) multiprocessing, all the better!

@shivaram
Copy link
Collaborator

It looks like there are some specific new features it adds (from the readme below). If we dont use these features we should use the builtin one

Objects can be transferred between processes using pipes or multi-producer/multi-consumer queues.

Objects can be shared between processes using a server process or (for simple data) shared memory.

Equivalents of all the synchronization primitives in threading are available.

A Pool class makes it easy to submit tasks to a pool of worker processes.

@ericmjonas
Copy link
Collaborator Author

It's not clear to me that these are features only of multiprocess, and the documentation doesn't make that clear either. For example, multiprocessING has a Pool https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers

@shivaram
Copy link
Collaborator

I think we are both agreeing on the point that we use the inbuilt library until we run out of features from it :) You can probably edit this issue title to be 'Use multiprocessing from standard library' ?

@shivaram shivaram added this to the v0.2 milestone Mar 9, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants