Should pathos provide a Queue?

[mmckerns]

Should pathos also provide a Queue that can be used with asynchronous apipe?
Something like: http://stackoverflow.com/questions/29631084/python-multiprocessing-in-objects, or alternatively working with a global Queue.

A Queue is the foundation of many 'non-pool' algorithms, and it wouldn't take too much to build one on top of an asynchronous pipe.

[rueberger]

Any updates on this? Recently ran into some problems with pickling using multiprocessing.Queue. It would be really nice if the pathos API transparently provided a Queue that leverages dill. Judging from that stackoverflow, it seems like this wouldn't be very hard?

[janssen]

I've switched over to using multiprocess and dill. I'd very much like to be able to deploy my system on a cluster, but I currently use multiprocess.Queue and multiprocess.Condition for signalling between agents. Will those work in pathos, as well?

[mmckerns]

@janssen: Yes, multiprocess has a Queue, and it is available inside of pathos.

>>> from pathos.helpers import mp as multiprocess
>>> print(multiprocess.Queue)
<function Queue at 0x10a69b5f0>

I'll close this if it's determined that multiprocess.Queue meets the requirements of this ticket.

[paulegradie]

Thanks @mmckerns for writing this incredibly useful package.

For anyone who uses this inside of a class and comes across the following error:
RuntimeError: Queue objects should only be shared between processes through inheritance

A way around this was discussed here:
https://stackoverflow.com/questions/9908781/sharing-a-result-queue-among-several-processes

In summary, you can also use the pathos multiprocessing.Manager.

    from pathos.helpers import mp as pathos_multiprocess
    manager = pathos_multiprocess.Manager()
    queue = manager.Queue()

[fny]

The multiprocess queue can't handle certain objects it seems? I created a custom result class which it can't pickle. 😢

[mmckerns]

@fny: multiprocess will use dill if it is installed. You can use dill.settings to try different serialization variants, but if none of the dill variants can serialize your custom object, then multiprocess will fail to handle the object. If your object is a custom class that you have built, you can, however, add a __reduce__ method to your custom class to ensure the object can be serialized.

[mmckerns]

I'm going to close this as worksforme