Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
344 lines (271 sloc) 13.6 KB

:mod:`multiprocessing.shared_memory` --- Provides shared memory for direct access across processes

.. module:: multiprocessing.shared_memory
   :synopsis: Provides shared memory for direct access across processes.

Source code: :source:`Lib/multiprocessing/shared_memory.py`

.. versionadded:: 3.8

.. index::
   single: Shared Memory
   single: POSIX Shared Memory
   single: Named Shared Memory


This module provides a class, :class:`SharedMemory`, for the allocation and management of shared memory to be accessed by one or more processes on a multicore or symmetric multiprocessor (SMP) machine. To assist with the life-cycle management of shared memory especially across distinct processes, a :class:`~multiprocessing.managers.BaseManager` subclass, :class:`SharedMemoryManager`, is also provided in the multiprocessing.managers module.

In this module, shared memory refers to "System V style" shared memory blocks (though is not necessarily implemented explicitly as such) and does not refer to "distributed shared memory". This style of shared memory permits distinct processes to potentially read and write to a common (or shared) region of volatile memory. Processes are conventionally limited to only have access to their own process memory space but shared memory permits the sharing of data between processes, avoiding the need to instead send messages between processes containing that data. Sharing data directly via memory can provide significant performance benefits compared to sharing data via disk or socket or other communications requiring the serialization/deserialization and copying of data.

Creates a new shared memory block or attaches to an existing shared memory block. Each shared memory block is assigned a unique name. In this way, one process can create a shared memory block with a particular name and a different process can attach to that same shared memory block using that same name.

As a resource for sharing data across processes, shared memory blocks may outlive the original process that created them. When one process no longer needs access to a shared memory block that might still be needed by other processes, the :meth:`close()` method should be called. When a shared memory block is no longer needed by any process, the :meth:`unlink()` method should be called to ensure proper cleanup.

name is the unique name for the requested shared memory, specified as a string. When creating a new shared memory block, if None (the default) is supplied for the name, a novel name will be generated.

create controls whether a new shared memory block is created (True) or an existing shared memory block is attached (False).

size specifies the requested number of bytes when creating a new shared memory block. Because some platforms choose to allocate chunks of memory based upon that platform's memory page size, the exact size of the shared memory block may be larger or equal to the size requested. When attaching to an existing shared memory block, the size parameter is ignored.

.. method:: close()

   Closes access to the shared memory from this instance.  In order to
   ensure proper cleanup of resources, all instances should call
   ``close()`` once the instance is no longer needed.  Note that calling
   ``close()`` does not cause the shared memory block itself to be
   destroyed.

.. method:: unlink()

   Requests that the underlying shared memory block be destroyed.  In
   order to ensure proper cleanup of resources, ``unlink()`` should be
   called once (and only once) across all processes which have need
   for the shared memory block.  After requesting its destruction, a
   shared memory block may or may not be immediately destroyed and
   this behavior may differ across platforms.  Attempts to access data
   inside the shared memory block after ``unlink()`` has been called may
   result in memory access errors.  Note: the last process relinquishing
   its hold on a shared memory block may call ``unlink()`` and
   :meth:`close()` in either order.

.. attribute:: buf

   A memoryview of contents of the shared memory block.

.. attribute:: name

   Read-only access to the unique name of the shared memory block.

.. attribute:: size

   Read-only access to size in bytes of the shared memory block.

The following example demonstrates low-level use of :class:`SharedMemory` instances:

>>> from multiprocessing import shared_memory
>>> shm_a = shared_memory.SharedMemory(create=True, size=10)
>>> type(shm_a.buf)
<class 'memoryview'>
>>> buffer = shm_a.buf
>>> len(buffer)
10
>>> buffer[:4] = bytearray([22, 33, 44, 55])  # Modify multiple at once
>>> buffer[4] = 100                           # Modify single byte at a time
>>> # Attach to an existing shared memory block
>>> shm_b = shared_memory.SharedMemory(shm_a.name)
>>> import array
>>> array.array('b', shm_b.buf[:5])  # Copy the data into a new array.array
array('b', [22, 33, 44, 55, 100])
>>> shm_b.buf[:5] = b'howdy'  # Modify via shm_b using bytes
>>> bytes(shm_a.buf[:5])      # Access via shm_a
b'howdy'
>>> shm_b.close()   # Close each SharedMemory instance
>>> shm_a.close()
>>> shm_a.unlink()  # Call unlink only once to release the shared memory

The following example demonstrates a practical use of the :class:`SharedMemory` class with NumPy arrays, accessing the same numpy.ndarray from two distinct Python shells:

A subclass of :class:`~multiprocessing.managers.BaseManager` which can be used for the management of shared memory blocks across processes.

A call to :meth:`~multiprocessing.managers.BaseManager.start` on a :class:`SharedMemoryManager` instance causes a new process to be started. This new process's sole purpose is to manage the life cycle of all shared memory blocks created through it. To trigger the release of all shared memory blocks managed by that process, call :meth:`~multiprocessing.managers.BaseManager.shutdown()` on the instance. This triggers a :meth:`SharedMemory.unlink()` call on all of the :class:`SharedMemory` objects managed by that process and then stops the process itself. By creating SharedMemory instances through a SharedMemoryManager, we avoid the need to manually track and trigger the freeing of shared memory resources.

This class provides methods for creating and returning :class:`SharedMemory` instances and for creating a list-like object (:class:`ShareableList`) backed by shared memory.

Refer to :class:`multiprocessing.managers.BaseManager` for a description of the inherited address and authkey optional input arguments and how they may be used to connect to an existing SharedMemoryManager service from other processes.

.. method:: SharedMemory(size)

   Create and return a new :class:`SharedMemory` object with the
   specified ``size`` in bytes.

.. method:: ShareableList(sequence)

   Create and return a new :class:`ShareableList` object, initialized
   by the values from the input ``sequence``.

The following example demonstrates the basic mechanisms of a :class:`SharedMemoryManager`:

The following example depicts a potentially more convenient pattern for using :class:`SharedMemoryManager` objects via the :keyword:`with` statement to ensure that all shared memory blocks are released after they are no longer needed:

When using a :class:`SharedMemoryManager` in a :keyword:`with` statement, the shared memory blocks created using that manager are all released when the :keyword:`with` statement's code block finishes execution.

The following example demonstrates basic use of a :class:`ShareableList` instance:

>>> from multiprocessing import shared_memory
>>> a = shared_memory.ShareableList(['howdy', b'HoWdY', -273.154, 100, None, True, 42])
>>> [ type(entry) for entry in a ]
[<class 'str'>, <class 'bytes'>, <class 'float'>, <class 'int'>, <class 'NoneType'>, <class 'bool'>, <class 'int'>]
>>> a[2]
-273.154
>>> a[2] = -78.5
>>> a[2]
-78.5
>>> a[2] = 'dry ice'  # Changing data types is supported as well
>>> a[2]
'dry ice'
>>> a[2] = 'larger than previously allocated storage space'
Traceback (most recent call last):
  ...
ValueError: exceeds available storage for existing str
>>> a[2]
'dry ice'
>>> len(a)
7
>>> a.index(42)
6
>>> a.count(b'howdy')
0
>>> a.count(b'HoWdY')
1
>>> a.shm.close()
>>> a.shm.unlink()
>>> del a  # Use of a ShareableList after call to unlink() is unsupported

The following example depicts how one, two, or many processes may access the same :class:`ShareableList` by supplying the name of the shared memory block behind it:

>>> b = shared_memory.ShareableList(range(5))         # In a first process
>>> c = shared_memory.ShareableList(name=b.shm.name)  # In a second process
>>> c
ShareableList([0, 1, 2, 3, 4], name='...')
>>> c[-1] = -999
>>> b[-1]
-999
>>> b.shm.close()
>>> c.shm.close()
>>> c.shm.unlink()
You can’t perform that action at this time.