Skip to content
Pydra Dataflow Engine
Jupyter Notebook Python Other
Branch: master
Clone or download
djarecka Merge pull request #153 from djarecka/shell_out_spec
customized input_spec and output_spec for ShellTask
Latest commit e734a9c Nov 10, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.
pydra modifying hash_file: adding option for tasks with a splitter over lis… Nov 11, 2019
tools MAINT: Add script to keep min-requirements synced with setup.cfg Jul 29, 2019
tutorial removing splitter and combiner kwargs from split/combine Aug 19, 2019
.gitattributes MAINT: Versioneer Oct 9, 2018
.gitignore adding tutorial notebooks Jul 26, 2019
.pre-commit-config.yaml update pre-commit Nov 9, 2019
.travis.yml small changes in travis Nov 9, 2019
LICENSE maint: license + pr template Jul 19, 2019 fix: add schema to setup process Dec 9, 2018
Makefile FIX: Specify black options in pyproject.toml Jul 17, 2019 add: codecov badge Aug 6, 2019
codecov.yml cov: add files to ignore Aug 6, 2019
long_description.rst MAINT: Move to setup.cfg + pyproject.toml Jul 17, 2019
min-requirements.txt MAINT: Add script to keep min-requirements synced with setup.cfg Jul 29, 2019
pydra_logo.png ADD logo to readme Oct 9, 2018
setup.cfg STY: black Jul 17, 2019 sty: black; remove workingdir Mar 14, 2019


A simple dataflow engine with scalable semantics.

Build Status codecov

The goal of pydra is to provide a lightweight Python dataflow engine for DAG construction, manipulation, and distributed execution.

Feature list:

  1. Python 3.7+ using type annotation and dataclasses
  2. Composable dataflows with simple node semantics. A dataflow can be a node of another dataflow.
  3. splitter and combiner provides many ways of compressing complex loop semantics
  4. Cached execution with support for a global cache across dataflows and users
  5. Distributed execution (presently via ConcurrentFutures and SLURM, more on their way)


This tutorial will walk you through the main concepts of Pydra! Binder

Please note that mybinder times out after an hour.


pip install pydra

Developer installation

Pydra requires Python 3.7+. To install in developer mode:

git clone
cd pydra
pip install -e .[dev]

It is also useful to install pre-commit:

pip install pre-commit
You can’t perform that action at this time.