A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin
- ActionChain - A workflow system for simple linear success/failure workflows.
- Airflow - Python-based workflow system created by AirBnb.
- Anduril - Component-based workflow framework for scientific data analysis.
- Antha - High-level language for biology.
- Bds - Scripting language for data pipelines.
- Bpipe - Tool for running and managing bioinformatics pipelines.
- Cluster flow - Command-line tool which uses common cluster managers to run bioinformatics pipelines.
- Compss - Programming model for distributed infrastructures.
- Conan2 - Light-weight workflow management application.
- Cosmos - Python library for massively parallel workflows.
- Flowr - Robust and efficient workflows using a simple language agnostic approach (R package).
- Dagobah - Simple DAG-based job scheduler in Python.
- Joblib - Set of tools to provide lightweight pipelining in Python.
- Luigi - Python module that helps you build complex pipelines of batch jobs.
- Makeflow - Workflow engine for executing large complex workflows on clusters.
- Mario - Scala library for defining data pipelines.
- Mistral - Python based workflow engine by the Open Stack project.
- Moa - Lightweight workflows in bioinformatics.
- Nextflow - Flow-based computational toolkit for reproducibile and scalable bioinformatics pipelines.
- NiPype - Workflows and interfaces for neuroimaging packages.
- OpenGE - Accelerated framework for manipulating and interpreting high-throughput sequencing data.
- PipEngine Ruby based launcher for complex biological pipelines.
- Pinball - Python based workflow engine by Pinterest.
- PyFlow - Lightweight parallel task engine.
- Pwrake - Parallel workflow extension for Rake
- Rabix - Python-based workflow toolkit based on the Common Workflow Language and Docker.
- Rmake - Wrapper for the creation of Makefiles, enabling massive parallelization.
- Rubra - Pipeline system for bioinformatics workflows.
- Ruffus - Computation Pipeline library for Python.
- Ruigi - Pipeline tool for R, inspired by Luigi.
- Sake - Self-documenting build automation tool.
- Scoop - Scalable Concurrent Operations in Python.
- Snakemake - Tool for running and managing bioinformatics pipelines.
- Spiff - Based on the Workflow Patterns initiative and implemented in Python.
- Swift - Fast easy parallel scripting - on multicores, clusters, clouds and supercomputers.
- Toil - Distributed pipeline workflow manager (mostly for genomics).
- Yap - Extensible parallel framework, written in Python using OpenMPI libraries.
- WorldMake - Easy Collaborative Reproducible Computing.
- ActivePapers - Computational science made reproducible and publishable.
- Apache Iravata - Framework for executing and managing computational workflows on distributed computing resources.
- Biokepler - Bioinformatics Scientific Workflow for Distributed Analysis of Large-Scale Biological Data.
- Chipster - Open source platform for data analysis.
- Galaxy - Web-based platform for biomedical research.
- Pegasus - Workflow Management System.
- Yabi - Online research environment for grid, HPC and cloud computing.
- Taverna - Domain independent workflow system.
- VisTrails - Scientific workflow and provenance management system.
- Wings - Semantic workflow system utilizing Pegasus as execution system.
- Common Workflow Language
- Cloudgene Workflow Language
- Workflow Definition Language
- Yet Another Workflow Language
- Workflow 4 Ever Initiative
- Workflow 4 Ever workflow research object model
- Workflow Patterns Initiative
- Workflow Patterns Library
- ResearchObject.org
- Beaker Notebook-style development environment.
- IPython A rich architecture for interactive computing.
- Jupyter Language-agnostic notebook literate programming environment.
- Pathomx - Interactive data workflows built on Python.
- Wakari - Web-based Python Data Analysis.
- Zeppelin - Web-based notebook that enables interactive data analytics.