parsl
set_stream_logger
set_file_logger
Apps are parallelized functions that execute independent of the control flow of the main python interpreter. We have two main types of Apps : PythonApps and BashApps. These are subclassed from AppBase.
This is the base class that defines the two external facing functions that an App must define. The __init__ () which is called when the interpreter sees the definition of the decorated function, and the __call__ () which is invoked when a decorated function is called by the user.
parsl.app.app.AppBase
Concrete subclass of AppBase that implements the Python App functionality.
parsl.app.python_app.PythonApp
Concrete subclass of AppBase that implements the Bash App functionality.
parsl.app.bash_app.BashApp
Futures are returned as proxies to a parallel execution initiated by a call to an App. We have two kinds of futures in Parsl: AppFutures and DataFutures.
parsl.dataflow.futures.AppFuture
parsl.app.futures.DataFuture
parsl.app.errors.ParslError
parsl.app.errors.NotFutureError
parsl.app.errors.InvalidAppTypeError
parsl.app.errors.AppException
parsl.app.errors.AppBadFormatting
parsl.app.errors.AppFailure
parsl.app.errors.MissingOutputs
parsl.app.errors.DependencyError
parsl.dataflow.error.DataFlowException
parsl.dataflow.error.DuplicateTaskError
parsl.dataflow.error.MissingFutError
parsl.dataflow.dflow.DataFlowKernel
Executors are abstractions that represent available compute resources to which you could submit arbitrary App tasks. An executor initialized with an Execution Provider can dynamically scale with the resources requirements of the workflow.
We currently have thread pools for local execution, remote workers from ipyparallel for executing on high throughput systems such as campus clusters, and a Swift/T executor for HPC systems.
parsl.executors.base.ParslExecutor
parsl.executors.threads.ThreadPoolExecutor
parsl.executors.ipp.IPyParallelExecutor
parsl.executors.HighThroughputExecutor
parsl.executors.HighThroughputExecutor
parsl.executors.swift_t.TurbineExecutor
parsl.executors.swift_t.runner
Execution providers are responsible for managing execution resources that have a Local Resource Manager (LRM). For instance, campus clusters and supercomputers generally have LRMs (schedulers) such as Slurm, Torque/PBS, Condor and Cobalt. Clouds, on the other hand, have API interfaces that allow much more fine-grained composition of an execution environment. An execution provider abstracts these types of resources and provides a single uniform interface to them.
parsl.providers.provider_base.ExecutionProvider
parsl.providers.LocalProvider
parsl.providers.SlurmProvider
parsl.providers.CobaltProvider
parsl.providers.CondorProvider
parsl.providers.TorqueProvider
parsl.providers.GridEngineProvider
parsl.providers.AWSProvider
parsl.providers.GoogleCloudProvider
parsl.providers.KubernetesProvider
For certain resources such as campus clusters or supercomputers at research laboratories, resource requirements may require authentication. For instance, some resources may allow access to their job schedulers from only their login-nodes, which require you to authenticate on through SSH, GSI-SSH and sometimes even require two-factor authentication. Channels are simple abstractions that enable the ExecutionProvider component to talk to the resource managers of compute facilities. The simplest Channel, LocalChannel, simply executes commands locally on a shell, while the SshChannel authenticates you to remote systems.
parsl.channels.channel_base.Channel
parsl.channels.LocalChannel
parsl.channels.SSHChannel
parsl.channels.SSHInteractiveLoginChannel
An execution provider is basically an adapter to various types of execution resources. The providers abstract away the interfaces provided by various systems to request, monitor, and cancel computate resources.
parsl.execution_provider_base.ExecutionProvider
parsl.providers.slurm.slurm.Slurm
parsl.providers.cobalt.cobalt.Cobalt
parsl.providers.condor.condor.Condor
parsl.providers.torque.torque.Torque
parsl.providers.local.local.Local
parsl.providers.aws.aws.EC2Provider
For certain resources such as campus clusters or supercomputers at research laboratories, resource requirements may require authentication. For instance some resources may allow access to their job schedulers from only their login-nodes which require you to authenticate on through SSH, GSI-SSH and sometimes even require two factor authentication. Channels are simple abstractions that enable the ExecutionProvider component to talk to the resource managers of compute facilities. The simplest Channel, LocalChannel simply executes commands locally on a shell, while the SshChannel authenticates you to remote systems.
parsl.channels.channel_base.Channel
parsl.channels.local.local.LocalChannel
parsl.channels.ssh.ssh.SshChannel
parsl.channels.ssh_il.ssh_il.SshILChannel
Launchers are basically wrappers for user submitted scripts as they are submitted to a specific execution resource.
parsl.launchers.SimpleLauncher
parsl.launchers.SingleNodeLauncher
parsl.launchers.AprunLauncher
parsl.launchers.SrunLauncher
parsl.launchers.SrunMPILauncher
This section deals with functionality related to controlling the flow of tasks to various executors.
parsl.dataflow.flow_control.FlowControl
parsl.dataflow.flow_control.FlowNoControl
parsl.dataflow.flow_control.Timer
Strategies are responsible for tracking the compute requirements of a workflow as it is executed and scaling the resources to match it.
parsl.dataflow.strategy.Strategy
parsl.dataflow.memoization.Memoizer