Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why do we use multiprocess instead of multiprocessing #60

Closed
ericmjonas opened this issue Feb 23, 2017 · 5 comments
Closed

Why do we use multiprocess instead of multiprocessing #60

ericmjonas opened this issue Feb 23, 2017 · 5 comments
Milestone

Comments

@ericmjonas
Copy link
Collaborator

@ericmjonas ericmjonas commented Feb 23, 2017

This seems like an unnecessary dependency given that we just 1. use it in the stand-alone mode and 2. I think multiprocessing provides all the same functionality? I'm still not sure what the multiprocess fork actually does.

@shivaram
Copy link
Collaborator

@shivaram shivaram commented Feb 23, 2017

Just to clarify you mean what are the differences between multiprocess (https://github.com/uqfoundation/multiprocess) and multiprocessing (https://docs.python.org/2/library/multiprocessing.html) ?

@ericmjonas
Copy link
Collaborator Author

@ericmjonas ericmjonas commented Feb 23, 2017

Yes, exactly -- I think I depended on multiprocess out of habit / it used to be the only game in town, but if we can get away with the (built-in) multiprocessing, all the better!

@shivaram
Copy link
Collaborator

@shivaram shivaram commented Feb 23, 2017

It looks like there are some specific new features it adds (from the readme below). If we dont use these features we should use the builtin one

Objects can be transferred between processes using pipes or multi-producer/multi-consumer queues.

Objects can be shared between processes using a server process or (for simple data) shared memory.

Equivalents of all the synchronization primitives in threading are available.

A Pool class makes it easy to submit tasks to a pool of worker processes.

@ericmjonas
Copy link
Collaborator Author

@ericmjonas ericmjonas commented Feb 23, 2017

It's not clear to me that these are features only of multiprocess, and the documentation doesn't make that clear either. For example, multiprocessING has a Pool https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers

@shivaram
Copy link
Collaborator

@shivaram shivaram commented Feb 23, 2017

I think we are both agreeing on the point that we use the inbuilt library until we run out of features from it :) You can probably edit this issue title to be 'Use multiprocessing from standard library' ?

@shivaram shivaram added this to the v0.2 milestone Mar 9, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

2 participants
You can’t perform that action at this time.