New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why do we use multiprocess instead of multiprocessing #60

Closed
ericmjonas opened this Issue Feb 23, 2017 · 5 comments

Comments

Projects
None yet
2 participants
@ericmjonas
Collaborator

ericmjonas commented Feb 23, 2017

This seems like an unnecessary dependency given that we just 1. use it in the stand-alone mode and 2. I think multiprocessing provides all the same functionality? I'm still not sure what the multiprocess fork actually does.

@shivaram

This comment has been minimized.

Show comment
Hide comment
@shivaram

shivaram Feb 23, 2017

Collaborator

Just to clarify you mean what are the differences between multiprocess (https://github.com/uqfoundation/multiprocess) and multiprocessing (https://docs.python.org/2/library/multiprocessing.html) ?

Collaborator

shivaram commented Feb 23, 2017

Just to clarify you mean what are the differences between multiprocess (https://github.com/uqfoundation/multiprocess) and multiprocessing (https://docs.python.org/2/library/multiprocessing.html) ?

@ericmjonas

This comment has been minimized.

Show comment
Hide comment
@ericmjonas

ericmjonas Feb 23, 2017

Collaborator

Yes, exactly -- I think I depended on multiprocess out of habit / it used to be the only game in town, but if we can get away with the (built-in) multiprocessing, all the better!

Collaborator

ericmjonas commented Feb 23, 2017

Yes, exactly -- I think I depended on multiprocess out of habit / it used to be the only game in town, but if we can get away with the (built-in) multiprocessing, all the better!

@shivaram

This comment has been minimized.

Show comment
Hide comment
@shivaram

shivaram Feb 23, 2017

Collaborator

It looks like there are some specific new features it adds (from the readme below). If we dont use these features we should use the builtin one

Objects can be transferred between processes using pipes or multi-producer/multi-consumer queues.

Objects can be shared between processes using a server process or (for simple data) shared memory.

Equivalents of all the synchronization primitives in threading are available.

A Pool class makes it easy to submit tasks to a pool of worker processes.

Collaborator

shivaram commented Feb 23, 2017

It looks like there are some specific new features it adds (from the readme below). If we dont use these features we should use the builtin one

Objects can be transferred between processes using pipes or multi-producer/multi-consumer queues.

Objects can be shared between processes using a server process or (for simple data) shared memory.

Equivalents of all the synchronization primitives in threading are available.

A Pool class makes it easy to submit tasks to a pool of worker processes.

@ericmjonas

This comment has been minimized.

Show comment
Hide comment
@ericmjonas

ericmjonas Feb 23, 2017

Collaborator

It's not clear to me that these are features only of multiprocess, and the documentation doesn't make that clear either. For example, multiprocessING has a Pool https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers

Collaborator

ericmjonas commented Feb 23, 2017

It's not clear to me that these are features only of multiprocess, and the documentation doesn't make that clear either. For example, multiprocessING has a Pool https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers

@shivaram

This comment has been minimized.

Show comment
Hide comment
@shivaram

shivaram Feb 23, 2017

Collaborator

I think we are both agreeing on the point that we use the inbuilt library until we run out of features from it :) You can probably edit this issue title to be 'Use multiprocessing from standard library' ?

Collaborator

shivaram commented Feb 23, 2017

I think we are both agreeing on the point that we use the inbuilt library until we run out of features from it :) You can probably edit this issue title to be 'Use multiprocessing from standard library' ?

@shivaram shivaram added this to the v0.2 milestone Mar 9, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment