Why do we use multiprocess instead of multiprocessing

[ericmjonas]

This seems like an unnecessary dependency given that we just 1. use it in the stand-alone mode and 2. I think multiprocessing provides all the same functionality? I'm still not sure what the multiprocess fork actually does.

[shivaram]

Just to clarify you mean what are the differences between multiprocess (https://github.com/uqfoundation/multiprocess) and multiprocessing (https://docs.python.org/2/library/multiprocessing.html) ?

[ericmjonas]

Yes, exactly -- I think I depended on multiprocess out of habit / it used to be the only game in town, but if we can get away with the (built-in) multiprocessing, all the better!

[shivaram]

It looks like there are some specific new features it adds (from the readme below). If we dont use these features we should use the builtin one

Objects can be transferred between processes using pipes or multi-producer/multi-consumer queues.

Objects can be shared between processes using a server process or (for simple data) shared memory.

Equivalents of all the synchronization primitives in threading are available.

A Pool class makes it easy to submit tasks to a pool of worker processes.

[ericmjonas]

It's not clear to me that these are features only of multiprocess, and the documentation doesn't make that clear either. For example, multiprocessING has a Pool https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers

[shivaram]

I think we are both agreeing on the point that we use the inbuilt library until we run out of features from it :) You can probably edit this issue title to be 'Use multiprocessing from standard library' ?