multiprocessing.rst

Distributed multiprocessing.Pool

Ray supports running distributed python programs with the multiprocessing.Pool API using Ray Actors instead of local processes. This makes it easy to scale existing applications that use multiprocessing.Pool from a single node to a cluster.

Quickstart

To get started, first install Ray, then use ray.util.multiprocessing.Pool in place of multiprocessing.Pool. This will start a local Ray cluster the first time you create a Pool and distribute your tasks across it. See the Run on a Cluster section below for instructions to run on a multi-node Ray cluster instead.

from ray.util.multiprocessing import Pool

def f(index):
    return index

pool = Pool()
for result in pool.map(f, range(100)):
    print(result)

The full multiprocessing.Pool API is currently supported. Please see the multiprocessing documentation for details.

Warning

The context argument in the Pool constructor is ignored when using Ray.

Run on a Cluster

This section assumes that you have a running Ray cluster. To start a Ray cluster, please refer to the cluster setup instructions.

To connect a Pool to a running Ray cluster, you can specify the address of the head node in one of two ways:

• By setting the RAY_ADDRESS environment variable.
• By passing the ray_address keyword argument to the Pool constructor.

from ray.util.multiprocessing import Pool

# Starts a new local Ray cluster.
pool = Pool()

# Connects to a running Ray cluster, with the current node as the head node.
# Alternatively, set the environment variable RAY_ADDRESS="auto".
pool = Pool(ray_address="auto")

# Connects to a running Ray cluster, with a remote node as the head node.
# Alternatively, set the environment variable RAY_ADDRESS="<ip_address>:<port>".
pool = Pool(ray_address="<ip_address>:<port>")

You can also start Ray manually by calling ray.init() (with any of its supported configuration options) before creating a Pool.