Parsl workflows are developed completely independently from their execution environment. There are very many different execution environments in which Parsl programs and their apps can run, and many of these environments have multiple options of how those Parsl programs and apps run, which makes configuration somewhat complex, and also makes determining how to set up Parsl's configuration for a particular set of choices fairly complex, though we think the actual configuration itself is reasonable simple.
Parsl offers an extensible configuration model through which the execution environment and communication within that environment is configured. Parsl is configured using :class:`~parsl.config.Config` object. For more information, see the :class:`~parsl.config.Config` class documentation. The following shows how the configuration can be specified.
import parsl from parsl.config import Config from parsl.executors.threads import ThreadPoolExecutor config = Config( executors=[ThreadPoolExecutor()], lazy_errors=True ) parsl.load(config)
Configuration How-To and Examples:
Note
Please note that all configuration examples below require customization for your account, allocation, Python environment, etc.
The configuration provided to Parsl tells Parsl what resources to use to run the Parsl program and apps, and how to use them. Therefore it is important to carefully evaluate certain aspects of the Parsl program and apps, and the planned compute resources, to determine an ideal configuration match. These aspects are: 1) where the Parsl apps will execute; 2) how many nodes will be used to execute the apps, and how long the apps will run; 3) should the scheduler allocate multiple nodes at one time; and 4) where will the main parsl program run and how will it communicate with the apps.
Stepping through the following question should help you formulate a suitable configuration. In addition, examples for some specific configurations follow.
- Where would you like the apps in the Parsl program to run?
| Target | Executor | Provider |
|---|---|---|
| Laptop/Workstation |
|
LocalProvider |
| Amazon Web Services |
|
AWSProvider |
| Google Cloud |
|
GoogleCloudProvider |
| Slurm based cluster or supercomputer |
|
SlurmProvider |
| Torque/PBS based cluster or supercomputer |
|
TorqueProvider |
| Cobalt based cluster or supercomputer |
|
CobaltProvider |
| GridEngine based cluster or grid |
|
GridEngineProvider |
| Condor based cluster or grid |
|
CondorProvider |
| Kubernetes cluster |
|
KubernetesProvider |
- How many nodes will you use to run them? What task durations give good performance on different executors?
| Executor | Number of Nodes [*] | Task duration for good performance |
|---|---|---|
| ThreadPoolExecutor | 1 (Only local) | Any |
| LowLatencyExecutor | <=10 | 10ms+ |
| IPyParallelExecutor | <=128 | 50ms+ |
| HighThroughputExecutor | <=2000 | Task duration(s)/#nodes >= 0.01 longer tasks needed at higher scale |
| ExtremeScaleExecutor | >1000, <=8000 [†] | >minutes |
| [*] | We assume that each node has 32 workers. If there are fewer workers launched per node, a higher number of nodes could be supported. |
| [†] | 8000 nodes with 32 workers each totalling 256000 workers is the maximum scale at which we've tested the ExtremeScaleExecutor. |
Warning
IPyParallelExecutor will be deprecated as of Parsl v0.8.0, with HighThroughputExecutor as the recommended replacement.
3. If you are running on a cluster or supercomputer, will you request multiple nodes per batch (scheduler) job? (Here we use the term block to be equivalent to a batch job.)
nodes_per_block = 1 |
||
|---|---|---|
| Provider | Executor choice | Suitable Launchers |
| Systems that don't use Aprun | Any |
|
| Aprun based systems | Any |
|
nodes_per_block > 1 |
||
|---|---|---|
| Provider | Executor choice | Suitable Launchers |
| TorqueProvider | Any |
|
| CobaltProvider | Any |
|
| SlurmProvider | Any |
|
Note
If you are on a Cray system, you most likely need the AprunLauncher to launch workers unless you are on a native Slurm system like :ref:`configuring_nersc_cori`
4. Where will you run the main Parsl program, given that you already have determined where the apps will run? (This is needed to determine how to communicate between the Parsl program and the apps.)
| Parsl program location | App execution target | Suitable channel |
|---|---|---|
| Laptop/Workstation | Laptop/Workstation | LocalChannel |
| Laptop/Workstation | Cloud Resources | None |
| Laptop/Workstation | Clusters with no 2FA | SSHChannel |
| Laptop/Workstation | Clusters with 2FA | SSHInteractiveLoginChannel |
| Login node | Cluster/Supercomputer | LocalChannel |
The following snippet shows an example configuration for executing remotely on San Diego Supercomputer Center's Comet supercomputer. The example uses an SSHChannel to connect remotely to Comet, the SlurmProvider to interface with the Slurm scheduler used by Comet and the SrunLauncher to launch workers.
.. literalinclude:: ../../parsl/configs/comet_htex_multinode.py
The following snippet shows an example configuration for accessing NERSC's Cori supercomputer. This example uses the IPythonParallel executor and connects to Cori's Slurm scheduler. It uses a remote SSH channel that allows the IPythonParallel controller to be hosted on the script's submission machine (e.g., a PC). It is configured to request 2 nodes configured with 1 TaskBlock per node. Finally it includes override information to request a particular node type (Haswell) and to configure a specific Python environment on the worker nodes using Anaconda.
.. literalinclude:: ../../parsl/configs/cori_htex_local_multinode.py
The following snippet shows an example configuration for accessing TACC's Stampede2 supercomputer. This example uses theHighThroughput executor and connects to Stampede2's Slurm scheduler.
.. literalinclude:: ../../parsl/configs/stampede2_htex_multinode.py
The following snippet shows an example configuration for executing on Argonne Leadership Computing Facility's Theta supercomputer. This example uses the HighThroughputExecutor and connects to Theta's Cobalt scheduler using the CobaltProvider. This configuration assumes that the script is being executed on the login nodes of Theta.
.. literalinclude:: ../../parsl/configs/theta_local_htex_multinode.py
The following snippet shows an example configuration for executing on Argonne Leadership Computing Facility's Cooley analysis and visualization system. The example uses the HighThroughputExecutor and connects to Cooley's Cobalt scheduler using the CobaltProvider. This configuration assumes that the script is being executed on the login nodes of Theta.
.. literalinclude:: ../../parsl/configs/cooley_htex_multinode.py
The following snippet shows an example configuration for executing remotely on Blue Waters, a flagship machine at the National Center for Supercomputing Applications. The configuration assumes the user is running on a login node and uses the TorqueProvider to interface with the scheduler, and uses the AprunLauncher to launch workers.
.. literalinclude:: ../../parsl/configs/bluewaters_htex.py
The following snippet shows an example configuration for executing remotely on Swan, an XC50 machine hosted by the Cray Partner Network. The example uses an SSHChannel to connect remotely Swan, uses the TorqueProvider to interface with the scheduler and the AprunLauncher to launch workers on the machine
.. literalinclude:: ../../parsl/configs/swan_ipp_multinode.py
The snippet below shows an example configuration for executing from a login node on IN2P3's Computing Centre. The configuration uses the LocalProvider to run on a login node primarily to avoid GSISSH, which Parsl does not support yet. This system uses Grid Engine which Parsl interfaces with using the GridEngineProvider.
.. literalinclude:: ../../parsl/configs/cc_in2p3_local_single_node.py
This Midway cluster is a campus cluster hosted by the Research Computing Center at the University of Chicago. The snippet below shows an example configuration for executing remotely on Midway. The configuration assumes the user is running on a login node and uses the SlurmProvider to interface with the scheduler, and uses the SrunLauncher to launch workers.
.. literalinclude:: ../../parsl/configs/midway_htex_multinode.py
The Open Science Grid (OSG) is a national, distributed computing Grid spanning over 100 individual sites to provide tens of thousands of CPU cores. The snippet below shows an example configuration for executing remotely on OSG. The configuration uses the SSHChannel to connect remotely to OSG, uses the CondorProvider to interface with the scheduler.
.. literalinclude:: ../../parsl/configs/osg_ipp_multinode.py
Note
Please note that boto3 library is a requirement to use AWS with Parsl.
This can be installed via python3 -m pip install parsl[aws]
Amazon Web services is a commercial cloud service which allows you to rent a range of computers and other computing services. The snippet below shows an example configuration for provisioning nodes from the Elastic Compute Cloud (EC2) service. The first run would configure a Virtual Private Cloud and other networking and security infrastructure that will be re-used in subsequent runs. The configuration uses the AWSProvider to connect to AWS.
.. literalinclude:: ../../parsl/configs/ec2_htex_single_node.py
Any collection of compute nodes without a scheduler setup for task scheduling can be considered an ad-hoc cluster. Often these machines have a shared filesystem such as NFS or Lustre. In order to use these resources with Parsl, they need to set-up for password-less SSH access.
To use these ssh-accessible collection of nodes as an ad-hoc cluster, we create an executor for each node, using the LocalProvider with SSHChannel to identify the node by hostname. An example configuration follows.
.. literalinclude:: ../../parsl/configs/ad_hoc_htex.py
Note
Multiple blocks should not be assigned to each node when using the HighThroughputExecutor
Note
Load-balancing will not work properly with this approach. In future work, a dedicated provider that supports load-balancing will be implemented. You can follow progress on this work here.
For help constructing a configuration, you can click on class names such as :class:`~parsl.config.Config` or :class:`~parsl.executors.HighThroughputExecutor` to see the associated class documentation. The same documentation can be accessed interactively at the python command line via, for example:
>>> from parsl.config import Config >>> help(Config)