Dispatch your trivially parallizable jobs with sharedmem.
To cite sharedmem use the DOI below
Now also supports Python 3.
- sharedmem.empty creates numpy arrays shared by child processes.
- sharedmem.MapReduce dispatches work to child processes, allowing work functions defined in nested scopes.
- sharedmem.MapReduce.ordered and sharedmem.MapReduce.critical implements the equivelant concepts as OpenMP ordered and OpenMP critical sections.
- Exceptions are properly handled, including unpicklable exceptions. Unexpected death of child processes (Slaves) is handled in a graceful manner.
Functions and variables are inherited from a fork syscall and the copy-on-write mechanism, except sharedmem variables which are writable from both child processes or the main process. Pickability of objects is not a concern.
Usual limitations of fork do apply. sharedmem.MapReduce is easier to use than multiprocessing.Pool, at the cost of not supporting Windows.
For documentation, please refer to http://rainwoodman.github.io/sharedmem .
Here we provide two simple examples to illustrate the usage:
Although the global variables are delivered via copy-on-write fork, sharedmem relies on python's pickle module to send and recieve the return value of 'work' functions.
As a consequence, if the underlying library used by the work function returns objects that are not pickle friendly, then we will receive a corrupted object on the master process.
This can happen, for example if the underlyihng library returns an object that stores a raw pointer as an attribute. After unpickling the result on a new process, the raw pointer will point to an undefined memory region, and the master process will segfault as a result.
It is not as exotic as it sounds. We ran into this issue when interfacing sharedmem with cosmosis, which stores a raw pointer as an attribute:
The solution is to unpack the result of the work function, down to low level objects that are pickle friendly, and return those instead of the unfriendly high level object.