leverage shared memory

[mmckerns]

could leverage shared memory using ctypes

[erogol]

Same question for me

[mmckerns]

Yes, this is on the 'todo' list. Good thing is I have some toy code for this already - just not ready for g.p. use in the trunk. Should get to it post-release.

[mbelmadani]

Is it possible to leverage shared memory now? I have a issue very similar to OP from this thread: http://stackoverflow.com/questions/28740955/working-with-pathos-multiprocessing-tool-in-python-and

I have a dict that gets updated as part of a decorator and, like your example in the thread, ends up empty. Any new developments I might be able to use for this? Thanks, I really appreciate the effort going into pathos. Just started using it today and it's great!

[mmckerns]

Shared memory can be accessed from multiprocess.

>>> from pathos.helpers import mp as multiprocess
>>> a = multiprocess.Array('i', 2)
>>> a[:] = 1,2
>>> a
<SynchronizedArray wrapper for <multiprocess.sharedctypes.c_int_Array_2 object at 0x10e6cacb0>>
>>>

I think this is sufficient, unless there's a need for a shared memory map or some higher order construct.

[matanox]

@mmckerns thanks for pathos. does this mean that pathos solves or greatly alleviates the slow arguments passing angle of the original standard multiprocessing pool ― in ways leveraging shared memory or equivalently fast methods ― without the caller having to explicitly set up shared memory in their user code for that?

[mmckerns]

@matanster: Unfortunately, no. pathos collects multiple ways of executing code on parallel/distributed resources, and provides a unified interface to do so. Under the hood, when it performs parallel processing, it leverages multiprocess (which is a fork of multiprocessing that has better serialization than pickle). You can use shared memory in multiprocessing, in multiprocess, and in pathos. None provide a shared memory map, however, it's my intent to add one to pathos... I just have not gotten around to it yet...

pathos does also provide threading, through multiprocess.dummy, and other parallel methods.

[matanox]

Hey thanks a lot for letting me know.

I started writing something that encapsulates a function call with python's inspect and re-wires through shared memory arguments which are large numpy arrays.

Otherwise multiprocessing makes it a small journey to adapt user target functions to using shared memory by hand, mainly by riding the globals object through an initializer function to the pool ― which is required in every module housing a function that is a concurrent call target, a rather intrusive change to target functions.

multiprocessing.shared_memory.SharedMemory seems to avoid that part for the target function, which can then just request the name of the shared memory rather than get it through its globals. Or it can be auto-assigned by decorating the function. Perhaps if you only removed the need to use globals, that reduces the complexity for user code if it's soundly designed. If that's helpful instead of developing a shared memory kind of map.

wrapper functions combined with python's inspect.signature enable inspecting and rewiring argument bindings ...

I guess that Ray provides one example for a design for shared memory in the context of concurrent dispatch.

---

(Having mentioned numpy, some of numpy is actually concurrent as is, which goes against the idea of specializing concurrent dispatch for numpy).

[matanox]

I'm not sure how much this topic touches close to the core of the goals of pathos.

[mmckerns]

It sounds interesting. You might want to look at dill.source and ppft (both started as part of pathos) -- both provide extensions of inspect for a parallel computing context.

[matanox]

Thanks, it's useful to have these available as tested sub-components a la carte. The caching one too.

[matanox]

I think that by only simplifying the cache interface of SharedMemory a tiny bit with judicious wrapping, shared memory can become as simple to apply to a program as it gets, thus expediting many single-machine concurrency workloads involving large data by a significant margin.