A simple dataflow engine with scalable semantics.
The goal of pydra is to provide a lightweight Python dataflow engine for DAG construction, manipulation, and distributed execution.
- Python 3.7+ using type annotation and dataclasses
- Composable dataflows with simple node semantics. A dataflow can be a node of another dataflow.
combinerprovides many ways of compressing complex loop semantics
- Cached execution with support for a global cache across dataflows and users
- Distributed execution (presently via ConcurrentFutures and SLURM, more on their way)
Please note that mybinder times out after an hour.
pip install pydra
Pydra requires Python 3.7+. To install in developer mode:
git clone firstname.lastname@example.org:nipype/pydra.git cd pydra pip install -e .[dev]
It is also useful to install pre-commit:
pip install pre-commit pre-commit