Skip to content

Existing Workflow systems

Stian Soiland-Reyes edited this page Nov 12, 2021 · 303 revisions

Permalink: https://s.apache.org/existing-workflow-systems

Cite as (update dates):

Peter Amstutz, Maxim Mikheev, Michael R. Crusoe, Neboj拧a Tijani膰, Samuel Lampa, et al. (2021): Existing Workflow systems. Common Workflow Language wiki, GitHub. https://s.apache.org/existing-workflow-systems updated 2021-09-13, accessed 2021-09-16.

Computational Data Analysis Workflow Systems

An incomplete list

Please add new entries at the bottom.

See also: https://github.com/pditommaso/awesome-pipeline

  1. Arvados - CWL-based distributed computing platform for data analysis on massive data sets. https://arvados.org/ https://github.com/arvados/arvados
  2. Apache Taverna http://www.taverna.org.uk/ https://taverna.incubator.apache.org/
  3. Galaxy http://galaxyproject.org/
  4. SHIWA https://www.shiwa-workflow.eu/
  5. Apache Oozie https://oozie.apache.org/
  6. DNANexus https://wiki.dnanexus.com/API-Specification-v1.0.0/IO-and-Run-Specifications https://wiki.dnanexus.com/API-Specification-v1.0.0/Workflows-and-Analyses
  7. ~~BioDT~~ http://www.biodatomics.com/ (dead link)
  8. Agave http://agaveapi.co/live-docs/
  9. DiscoveryEnvironment http://www.iplantcollaborative.org/ci/discovery-environment
  10. Wings http://www.wings-workflows.org/
  11. Knime https://www.knime.org/
  12. make, rake, drake, ant, scons & many others. Software development relies heavily on tools to manage workflows related to compiling and packaging applications. For the most part these are file based and usually run on a single node, usually supporting parallel steps (make -j) and in some cases able to dispatch build steps to other machines (https://code.google.com/p/distcc/) https://github.com/Factual/drake
  13. Snakemake https://snakemake.github.io
  14. BPipe http://bpipe.org http://docs.bpipe.org/
  15. Ruffus https://github.com/cgat-developers/ruffus
  16. NextFlow http://nextflow.io
  17. Luigi. Python package that helps you build complex pipelines of batch jobs http://github.com/spotify/luigi https://luigi.readthedocs.io
  18. SciLuigi. Helper library built on top of Luigi to ease development of Scientific workflows in Luigi: http://github.com/pharmbio/sciluigi
  19. Luigi Analysis Workflow (LAW) https://github.com/riga/law
  20. GATK Queue https://www.broadinstitute.org/gatk/guide/topic?name=queue
  21. Yabi https://ccg.murdoch.edu.au/yabi
  22. seqware Workflows are written in Java and executed using the Oozie Workflow Engine on Hadoop or SGE clusters. Uses Zip64 files to group the workflow definition file, workflow itself, sample settings, and data dependencies in a single file that can be exchanged between SeqWare users or archived. https://seqware.github.io/ https://seqware.github.io/docs/6-pipeline/
  23. Ketrew https://github.com/hammerlab/ketrew
  24. Pegasus http://pegasus.isi.edu/
  25. Apache Airflow https://github.com/apache/airflow
  26. Couler https://github.com/couler-proj/couler - Unified interface for constructing and managing workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.
  27. Cosmos https://cosmos.hms.harvard.edu/documentation/index.html http://bioinformatics.oxfordjournals.org/content/early/2014/07/24/bioinformatics.btu385.full [paper] Cosmos2: https://github.com/LPM-HMS/COSMOS2 http://cosmos.hms.harvard.edu/COSMOS2/
  28. Pinball https://github.com/pinterest/pinball
  29. bcbio https://bcbio-nextgen.readthedocs.org/en/latest/
  30. Chronos https://github.com/mesos/chronos
  31. Azkaban https://azkaban.github.io/
  32. Apache NiFi https://nifi.apache.org/docs/nifi-docs/html/overview.html
  33. flowr (R-based) http://docs.flowr.space/ https://github.com/sahilseth/flowr
  34. Mistral https://github.com/arteria-project https://wiki.openstack.org/wiki/Mistral#What_is_Mistral.3F https://docs.openstack.org/mistral/latest/user/wf_lang_v2.html
  35. nipype http://nipy.org/nipype/
  36. End of Day https://github.com/joestubbs/endofday
  37. BioDSL https://github.com/maasha/BioDSL
  38. BigDataScript http://pcingola.github.io/BigDataScript/
  39. Omics Pipe: uses Ruffus http://sulab.scripps.edu/omicspipe/
  40. Ensembl Hive https://github.com/Ensembl/ensembl-hive
  41. QuickNGS http://bifacility.uni-koeln.de/quickngs/web
  42. GenePattern http://www.broadinstitute.org/cancer/software/genepattern/
  43. Chipster http://chipster.csc.fi/
  44. The Genome Modeling System https://github.com/genome/gms
  45. Cuneiform, A Functional Workflow Language https://github.com/joergen7/cuneiform http://www.cuneiform-lang.org/
  46. Anvaya http://www.ncbi.nlm.nih.gov/pubmed/22809419 http://webapp.cabgrid.res.in/biocomp/Anvaya/ANVAYA_Main.html#HOWTO_INSTALL_ANVAYA
  47. Makeflow http://ccl.cse.nd.edu/software/makeflow/
  48. Apache Airavata http://airavata.apache.org/
  49. Pyflow https://github.com/Illumina/pyflow
  50. Cluster Flow http://clusterflow.io
  51. Unipro UGENE http://ugene.net/ https://doi.org/10.7717/peerj.644
  52. CloudSlang http://www.cloudslang.io/
  53. Stacks http://catchenlab.life.illinois.edu/stacks/
  54. Leaf http://www.francesconapolitano.it/leaf/index.html
  55. omictools http://omictools.com/
  56. Job Description Language. The Job Description Language, JDL, is a high-level, user-oriented language based on Condor classified advertisements for describing jobs and aggregates of jobs such as Direct Acyclic Graphs and Collections. https://edms.cern.ch/ui/file/590869/1/WMS-JDL.pdf
  57. YAWL yet another workflow language https://doi.org/10.1016/j.is.2004.02.002 http://www.yawlfoundation.org/
  58. Triquetrum https://projects.eclipse.org/projects/technology.triquetrum https://github.com/eclipse/triquetrum/
  59. Kronos https://github.com/jtaghiyar/kronos
  60. qsubsec https://github.com/alastair-droop/qsubsec https://doi.org/10.1093/bioinformatics/btv698
  61. YesWorkflow http://yesworkflow.org
  62. gwf - Grid WorkFlow https://github.com/gwforg/gwf http://gwf.readthedocs.io/
  63. Fireworks. https://github.com/materialsproject/fireworks https://github.com/materialsproject/fireworks https://doi.org/10.1002/cpe.3505}
  64. NGLess: NGS with less work http://ngless.rtfd.io
  65. pypipegraph https://github.com/TyberiusPrime/pypipegraph
  66. Cromwell https://github.com/broadinstitute/cromwell
  67. Dagobah - Simple DAG-based job scheduler in Python. https://github.com/thieman/dagobah
  68. sushi https://github.com/uzh/sushi
  69. Clinical Trial Processor - A program for processing clinical trials data. http://mircwiki.rsna.org/index.php?title=MIRC_CTP
  70. Noodles http://nlesc.github.io/noodles/
  71. Swift & Swift/T http://swift-lang.org/main/ http://swift-lang.org/Swift-T/
  72. Consonance (runs SeqWare & CWL) https://github.com/Consonance/consonance/wiki
  73. Dog https://github.com/dogtools/dog
  74. Produce https://github.com/texttheater/produce
  75. LONI Pipeline http://pipeline.loni.usc.edu/
  76. Cpipe https://github.com/MelbourneGenomics/cpipe
  77. AWE https://github.com/MG-RAST/AWE
  78. (Py)COMPSs https://www.bsc.es/research-and-development/software-and-apps/software-list/comp-superscalar/
  79. KLIKO https://github.com/gijzelaerr/kliko
  80. SoS Workflow https://github.com/vatlab/SoS https://vatlab.github.io/sos-docs/ https://doi.org/10.1093/bioinformatics/bty405 https://doi.org/10.1371/journal.pcbi.1006843
  81. XNAT Pipeline Engine https://wiki.xnat.org/display/XNAT/Pipeline+Engine https://wiki.xnat.org/display/XNAT/XNAT+Pipeline+Development+Schema
  82. Metapipe https://github.com/TorkamaniLab/metapipe
  83. OCCAM (Open Curation for Computer Architecture Modeling) https://occam.cs.pitt.edu/
  84. Copernicus http://www.copernicus-computing.org
  85. iRODS Rule Language https://github.com/samuell/irods-cheatsheets/blob/master/irods-rule-lang-full-guide.md
  86. VisTrails https://www.vistrails.org
  87. Bionode Watermill https://github.com/bionode/bionode-watermill
  88. BIOVIA Pipeline Pilot Overview http://accelrys.com/products/collaborative-science/biovia-pipeline-pilot/
  89. Dagman A meta-scheduler for HTCondor https://research.cs.wisc.edu/htcondor/dagman/dagman.html
  90. UNICORE https://www.unicore.eu/docstore/workflow-7.6.0/workflow-manual.html#wf_dialect
  91. Toil (A scalable, efficient, cross-platform and easy-to-use workflow engine in pure Python) https://github.com/BD2KGenomics/toil
  92. Cylc (a workflow engine for cycling systems) https://cylc.github.io/
  93. Autodesk Cloud Compute Canon https://github.com/Autodesk/cloud-compute-cannon
  94. Civet https://github.com/TheJacksonLaboratory/civet
  95. Cumulus https://github.com/Kitware/cumulus
  96. High-performance integrated virtual environment (HIVE) https://hive.biochemistry.gwu.edu
  97. Cloudgene http://cloudgene.uibk.ac.at/cloudgene-yaml
  98. FASTR https://bitbucket.org/bigr_erasmusmc/fastr/ http://fastr.readthedocs.io/en/stable/
  99. BioMake https://github.com/evoldoers/biomake https://doi.org/10.1101/093245
  100. remake https://github.com/richfitz/remake
  101. SciFloware http://www-sop.inria.fr/members/Didier.Parigot/pmwiki/Scifloware/
  102. OpenAlea http://openalea.gforge.inria.fr/dokuwiki/doku.php https://hal.archives-ouvertes.fr/hal-01166298/file/openalea-PradalCohen-Boulakia.pdf
  103. COMBUSTI/O https://github.com/jarlebass/combustio http://hdl.handle.net/10037/9361
  104. BioCloud https://github.com/ccwang002/biocloud-server-kai https://doi.org/10.6342/NTU201601295
  105. Triana http://www.trianacode.org/ (website seems to have been taken over by SEO)
  106. Kepler https://kepler-project.org/
  107. Anduril http://anduril.org/site/
  108. dgsh http://www.dmst.aueb.gr/dds/sw/dgsh/
  109. EDGE bioinformatics: Empowering the Development of Genomics Expertise https://bioedge.lanl.gov/edge_ui/ http://edge.readthedocs.io/ https://lanl-bioinformatics.github.io/EDGE/
  110. Pachyderm http://pachyderm.io/ http://pachyderm.readthedocs.io/en/stable/advanced/advanced.html
  111. Digdag https://www.digdag.io/
  112. Agua / Automated Genomics Utilities Agent http://aguadev.org
  113. BioDepot Workflow Builder (BwB) https://github.com/BioDepot/BioDepot-workflow-builder https://doi.org/10.1101/099010
  114. IMP: a pipeline for reproducible reference-independent integrated metagenomic and metatranscriptomic analyses http://r3lab.uni.lu/web/imp/ https://doi.org/10.1186/s13059-016-1116-8
  115. Butler https://github.com/llevar/butler
  116. adage / yadage https://github.com/diana-hep/adage https://github.com/diana-hep/yadage
  117. HI-WAY: Execution of Scientific Workflows on Hadoop YARN https://github.com/marcbux/Hi-WAY https://openproceedings.org/2017/conf/edbt/paper-248.pdf
  118. OpenMOLE https://github.com/openmole/openmole https://www.openmole.org/ https://doi.org/10.3389/fninf.2017.00021
  119. Biopet https://github.com/biopet/biopet
  120. Nephele https://nephele.niaid.nih.gov/
  121. TOPPAS https://doi.org/10.1021/pr300187f
  122. SBpipe https://pdp10.github.io/sbpipe/ https://github.com/pdp10/sbpipe https://doi.org/10.1186/s12918-017-0423-3
  123. Dray http://dray.it/ (website seems to have been taken over by SEO)
  124. GenomeVIP https://github.com/ding-lab/GenomeVIP https://doi.org/10.1101/gr.211656.116
  125. GridSAM https://sourceforge.net/projects/gridsam/
  126. Roddy https://github.com/eilslabs/Roddy
  127. SciFlo (historical; doesn't seem to be maintained anymore) https://web.archive.org/web/20161118011409/https://sciflo.jpl.nasa.gov/SciFloWiki/FrontPage
  128. GNU Guix Workflow Language https://git.roelj.com/guix/gwl.git#gnu-guix-workflow-language-extension https://github.com/UMCUGenetics/guix-workflows/blob/master/umcu/workflows/rnaseq.scm
  129. Porcupine https://timvanmourik.github.io/Porcupine/
  130. Parsl (a Parallel Scripting Library for Python) http://parsl-project.org
  131. ECFLOW (Workflow primarily for Meteorological Applications) https://software.ecmwf.int/wiki/display/ECFLOW/ecflow+home
  132. Ophidia http://ophidia.cmcc.it/
  133. WebLicht https://weblicht.sfs.uni-tuebingen.de/
  134. GATE Cloud https://cloud.gate.ac.uk/
  135. SCIPION http://scipion.cnb.csic.es/m/home/ https://github.com/I2PC/scipion/wiki/Creating-a-Protocol
  136. Ergatis http://ergatis.sourceforge.net/
  137. TIGR "Workflow" https://sourceforge.net/projects/tigr-workflow/ http://tigr-workflow.sourceforge.net/
  138. Archivematica https://wiki.archivematica.org/Main_Page (A preservation workflow system that implements the ISO-OAIS standard using gearman/MCP)
  139. Martian http://martian-lang.org/about/
  140. BioMAJ http://genouest.github.io/biomaj/
  141. Conveyor http://conveyor.cebitec.uni-bielefeld.de (retired). https://academic.oup.com/bioinformatics/article/27/7/903/230562/Conveyor-a-workflow-engine-for-bioinformatic
  142. Biopipe http://www.biopipe.org (appears to be defunct) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC403782/
  143. Wildfire http://wildfire.bii.a-star.edu.sg/ https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-6-69
  144. BioWBI http://bioinformatics.hsanmartino.it/bits_library/library/00079.pdf
  145. BioWMS http://bioinformatics.hsanmartino.it/bits_library/library/00568.pdf
  146. BioMoby http://biomoby.open-bio.org/ https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-7-523
  147. SIBIOS http://ieeexplore.ieee.org/document/1309094/
  148. NGSANE https://github.com/BauerLab/ngsane https://academic.oup.com/bioinformatics/article/30/10/1471/266879/NGSANE-a-lightweight-production-informatics
  149. Pwrake https://github.com/misshie/Workflows https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3180464/
  150. Nesoni https://github.com/Victorian-Bioinformatics-Consortium/nesoni
  151. Skam http://skam.sourceforge.net/skam-intro.html
  152. TREVA http://bioinformatics.petermac.org/treva/ http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0095217
  153. EGene https://www.semanticscholar.org/paper/EGene-a-configurable-pipeline-generation-system-fo-Durham-Kashiwabara/4c0656195b5efcdd3aa7bdcb55fc95a957c150aa https://academic.oup.com/bioinformatics/article/30/18/2659/2475637/EuGene-PP-a-next-generation-automated-annotation
  154. WEP https://bioinformatics.cineca.it/wep/ https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-14-S7-S11
  155. Microbase http://www.microbasecloud.com/
  156. e-Science Central http://www.esciencecentral.co.uk/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3538293/
  157. Cyrille2 https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-9-96
  158. PaPy https://code.google.com/archive/p/papy/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3051902/
  159. JobCenter https://github.com/yeastrc/jobcenter https://scfbm.biomedcentral.com/articles/10.1186/1751-0473-7-8
  160. CoreFlow https://www.ncbi.nlm.nih.gov/pubmed/24503186
  161. dynamic-pipeline https://code.google.com/archive/p/dynamic-pipeline/
  162. XiP http://xip.hgc.jp/wiki/en/Main_Page https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3530915/
  163. Eoulsan http://www.outils.genomique.biologie.ens.fr/eoulsan/ https://www.ncbi.nlm.nih.gov/pubmed/22492314
  164. CloudDOE http://clouddoe.iis.sinica.edu.tw/
  165. BioPig https://github.com/JGI-Bioinformatics/biopig https://www.ncbi.nlm.nih.gov/pubmed/24021384
  166. SeqPig https://github.com/HadoopGenomics/SeqPig https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3866557/
  167. zymake http://www-personal.umich.edu/~ebreck/code/zymake/
  168. JMS https://github.com/RUBi-ZA/JMS http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0134273
  169. CLC Genomics Workbench https://www.qiagenbioinformatics.com/products/clc-genomics-workbench/
  170. NG6 http://ng6.toulouse.inra.fr/ https://www.ncbi.nlm.nih.gov/pubmed/22958229
  171. VIBE http://www.incogen.com/vibe/
  172. WDL (Workflow Description Language) https://github.com/broadinstitute/wdl
  173. SciFlow https://github.com/kaizhang/SciFlow (not to be confused with SciFloware and SciFlo).
  174. Bioshake https://github.com/PapenfussLab/bioshake
  175. SciPipe http://scipipe.org
  176. Kapacitor / TICKscripts https://docs.influxdata.com/kapacitor/v1.3//tick/
  177. AiiDA: Automated Interactive Infrastructure and Database for Computational Science http://www.aiida.net/ https://doi.org/10.1016/j.commatsci.2015.09.013
  178. Reflow: a language and runtime for distributed, integrated data processing in the cloud https://github.com/grailbio/reflow
  179. Resolwe: an open source dataflow package for Django framework https://github.com/genialis/resolwe
  180. Yahoo! Pipes (historical) https://en.wikipedia.org/wiki/Yahoo!_Pipes
  181. Walrus https://github.com/fjukstad/walrus
  182. Apache Beam https://beam.apache.org/
  183. CLOSHA https://closha.kobic.re.kr/ https://www.bioexpress.re.kr/go_tutorial http://docplayer.net/19700397-Closha-manual-ver1-1-kobic-korean-bioinformation-center-kogun82-kribb-re-kr-2016-05-08-bioinformatics-workflow-management-system-in-bio-express.html https://doi.org/10.1186/s12859-018-2019-3
  184. WopMars https://github.com/aitgon/wopmars http://wopmars.readthedocs.io/
  185. flowing-clj https://github.com/stain/flowing-clj
  186. Plumbing and Graph https://github.com/plumatic/plumbing
  187. LabView http://www.ni.com/en-us/shop/labview.html
  188. MyOpenLab http://myopenlab.org/
  189. Max/MSP https://cycling74.com/products/max/
  190. NoFlo https://noflojs.org/
  191. Flowstone http://www.dsprobotics.com/flowstone.html
  192. HyperLoom https://code.it4i.cz/ADAS/loom https://code.it4i.cz/ADAS/loom
  193. Dask http://dask.pydata.org/en/latest/ https://github.com/dask/dask
  194. Stimela https://github.com/SpheMakh/Stimela https://github.com/SpheMakh/Stimela/wiki https://www.acru.ukzn.ac.za/~cosmosafari2017/wp-content/uploads/2017/02/makhathini.pdf
  195. JTracker https://jtracker.io/ https://github.com/jtracker-io
  196. PipelineDog http://pipeline.dog/ https://github.com/zhouanbo/pipelinedog https://doi.org/10.1093/bioinformatics/btx759
  197. DALiuGE https://arxiv.org/abs/1702.07617 https://github.com/ICRAR/daliuge https://daliuge.readthedocs.io/
  198. Overseer https://github.com/framed-data/overseer
  199. Squonk https://squonk.it/
  200. GC3Pie https://github.com/uzh/gc3pie
  201. Fractalide https://github.com/fractalide/fractalide
  202. TOGGLe http://toggle.southgreen.fr/ https://doi.org/10.1101/245480
  203. Askalon http://www.askalon.org
  204. Eclipse ICE (The Integrated Computational Environment) https://www.eclipse.org/ice
  205. Sandia Analysis Workbench (SAW) http://www.sandia.gov/saw/
  206. dispel4py https://github.com/dispel4py/dispel4py
  207. Jobber https://pypi.python.org/pypi/Jobber/0.1.4
  208. NeatSeq-Flow http://neatseq-flow.readthedocs.io/
  209. S4M https://bitbucket.org/uqokorn/s4m_base/wiki/Home
  210. Loom http://med.stanford.edu/gbsc/loom.html https://github.com/StanfordBioinformatics/loom http://loom.readthedocs.io/en/latest/templates.html
  211. Watchdog - XML Schema-based bioinformatics workflow system https://www.bio.ifi.lmu.de/software/watchdog/ https://doi.org/10.1186/s12859-018-2107-4 https://github.com/klugem/watchdog
  212. phpflo https://github.com/phpflo/phpflo
  213. BASTet: Berkeley Analysis and Storage Toolkit https://openmsi.nersc.gov/openmsi/client/bastet.html https://biorack.github.io/BASTet/ https://doi.org/10.1109/TVCG.2017.2744479
  214. Tavaxy: Pattern based workflow system for the bioinformatics domain http://www.tavaxy.org/
  215. Ginflow: Decentralised adaptive workflow engine https://ginflow.inria.fr/
  216. SciApps: A cloud-based platform for reproducible bioinformatics workflows https://doi.org/10.1093/bioinformatics/bty439 https://www.sciapps.org/
  217. Stoa: Script Tracking for Observational Astronomy https://github.com/petehague/Stoa
  218. Collective Knowledge (CK) framework http://cknowledge.org/
  219. QosCosGrid (QCG) http://www.qoscosgrid.org/ http://www.qoscosgrid.org/trac/qcg-broker/wiki/qcg-advanced-client%20
  220. High-Throughput Binding Affinity Calculator (HTBAC) https://htbac.readthedocs.io/en/latest/ https://github.com/radical-cybertools/htbac https://arxiv.org/abs/1801.01174
  221. BioWorkbench (Swift-based) https://arxiv.org/abs/1801.03915 https://github.com/mmondelli/bioworkbench
  222. ENVI Task Engine https://gbdxdocs.digitalglobe.com/docs/envi-task-engine https://www.harrisgeospatial.com/Learn/Whitepapers/TabId/2359/ArtMID/10212/ArticleID/17299/Workflow-Tools-in-ENVI.aspx https://envi-py-engine.readthedocs.io/en/latest/index.html
  223. Pypeline https://github.com/cgarciae/pypeln
  224. mpipe http://vmlaker.github.io/mpipe/
  225. idseq-dag https://github.com/chanzuckerberg/idseq-dag
  226. Piper (based upon GATK Queue) https://github.com/NationalGenomicsInfrastructure/piper
  227. Apache Object Oriented Data Technology (OODT) http://oodt.apache.org/
  228. JX Workflow (DSL for Makeflow) https://ccl.cse.nd.edu/software/manuals/jx-quick.html http://ccl.cse.nd.edu/research/papers/jx-escience-2018.pdf
  229. The Adaptable IO System (ADIOS), ADIOS using applications can be the orchestrated into a workflow http://csmd.ornl.gov/adios
  230. GenPipes https://bitbucket.org/mugqic/genpipes https://doi.org/10.1101/459552
  231. Argo https://argoproj.github.io/ https://github.com/argoproj/argo https://github.com/argoproj/argo/blob/master/examples/README.md
  232. Reana https://reana.readthedocs.io/en/latest/ https://github.com/reanahub/reana
  233. Cuisine Framework https://www.astron.nl/~renting/cuisine.html
  234. Niassa https://github.com/oicr-gsi/niassa https://oicr-gsi.github.io/niassa-docs/
  235. pypeFLOW https://github.com/PacificBiosciences/pypeFLOW
  236. Tiny Cloud Engine http://ka.cb.k.u-tokyo.ac.jp/tce/
  237. Xbowflow https://github.com/ChrisSuess/Project-Xbow/tree/master/xbowflow
  238. AdaptiveMd https://github.com/markovmodel/adaptivemd
  239. Meshroom https://github.com/alicevision/meshroom
  240. LSST Data Management https://github.com/lsst/pipe_base
  241. CGAT-core https://github.com/cgat-developers/cgat-core
  242. Prefect https://docs.prefect.io/
  243. Apache SCXML engine https://commons.apache.org/proper/commons-scxml/guide/core-engine.html https://commons.apache.org/proper/commons-scxml/guide/scxml-documents.html
  244. IceProd https://github.com/WIPACrepo/iceprod
  245. AnADAMA2 http://huttenhower.sph.harvard.edu/anadama2
  246. Luna https://luna-lang.org/
  247. Passerelle https://code.google.com/archive/a/eclipselabs.org/p/passerelle
  248. Kurator-Akka https://github.com/kurator-org/kurator-akka
  249. Jug https://doi.org/10.5334/jors.161
  250. Node-RED https://nodered.org/
  251. Databolt Flow https://github.com/d6t/d6tflow
  252. Frictionless Data Package Pipelines https://github.com/frictionlessdata/datapackage-pipelines
  253. DataFlows https://github.com/datahq/dataflows
  254. Volcano https://github.com/volcano-sh/volcano
  255. DataJoint https://datajoint.io/
  256. DIRAC3 (LHCb grid software) https://doi.org/10.1088/1742-6596/219/6/062029
  257. Orange https://orange.biolab.si/ http://jmlr.org/papers/v14/demsar13a.html
  258. Ensemble Toolkit (EnTK) https://radicalentk.readthedocs.io/en/latest/entk.html https://arxiv.org/abs/1602.00678v2
  259. BioQueue http://www.bioqueue.org/ https://github.com/liyao001/BioQueue https://doi.org/10.1093/bioinformatics/btx403
  260. mlr3pipelines https://mlr3pipelines.mlr-org.com/
  261. Kedro - Production-Ready Data & ML Pipelines https://github.com/quantumblacklabs/kedro
  262. DATAVIEW - DATAVIEW is a big data workflow management system. https://github.com/shiyonglu/DATAVIEW
  263. SecDATAVIEW - SecDATAVIEW is a security oriented big data workflow management system compatible with the heterogeneous computing environments. https://github.com/shiyonglu/SecDATAVIEW
  264. Giraffe Tools https://giraffe.tools/
  265. CERAMICCA - Cloud Engine Resource for Accelerated Medical Image Computing for Clinical Applications: https://ceramicca.ensc.sfu.ca
  266. uap - Universal Analysis Pipeline. A workflow management system dedicated to robust, consistent, and reproducible HTS data analysis. https://github.com/yigbt/uap
  267. signac https://signac.io/
  268. cwltool - reference implementation of Common Workflow Language (CWL) https://github.com/common-workflow-language/cwltool/
  269. CWLEXEC - CWL executor for IBM Spectrum LSF clusters https://github.com/IBMSpectrumComputing/cwlexec
  270. drmr https://github.com/ParkerLab/drmr/ https://drmr.readthedocs.io/
  271. Autosubmit - a versatile tool to manage Weather and Climate Experiments in diverse Supercomputing Environments. https://www.bsc.es/research-and-development/software-and-apps/software-list/autosubmit https://autosubmit.readthedocs.io/en/latest/
  272. JUDI https://doi.org/10.1093/bioinformatics/btz956 https://github.com/ncbi/JUDI
  273. Sumatra - a tool for managing and tracking projects based on numerical simulation or analysis, with the aim of supporting reproducible research https://neuralensemble.org/sumatra/
  274. Netflix Conductor: https://netflix.github.io/conductor/
  275. Pipengine https://doi.org/10.21105/joss.00341 https://github.com/fstrozzi/bioruby-pipengine
  276. MyQueue https://doi.org/10.21105/joss.01844 https://gitlab.com/myqueue/myqueue
  277. Drake R package https://doi.org/10.21105/joss.00550 https://github.com/ropensci/drake (superseded by targets)
  278. MaDaTS https://10.21105/joss.00830 https://github.com/dghoshal-lbl/madats
  279. Cadence https://cadenceworkflow.io/
  280. Merlin https://merlin.readthedocs.io/
  281. Janis https://janis.readthedocs.io/en/latest/index.html
  282. AlphaSQL https://github.com/Matts966/alphasql
  283. zeebe https://github.com/zeebe-io/zeebe
  284. durabletask https://github.com/Azure/durabletask
  285. Illumina State Language https://iap-docs.readme.io/docs/wes_illumina-states-language
  286. Moteur http://www.i3s.unice.fr/~johan/publis/MOTEUR-poster-A4.pdf https://indico.cern.ch/event/286/contributions/654974/attachments/530521/731563/userForumEGEE_MOTEUR.pdf
  287. SimStack http://www.simstack.de/
  288. Maestro Workflow Conductor https://github.com/LLNL/maestrowf
  289. HyWare https://doi.org/10.1007/s41060-020-00237-x
  290. HyperFlow https://github.com/hyperflow-wms/
  291. BRANE Framework https://onnovalkering.github.io/brane/
  292. ApolloWF https://apollowf.github.io/
  293. IS-EPOS Platform https://ieeexplore.ieee.org/document/9308147 https://tcs.ah-epos.eu/
  294. pyinvoke http://www.pyinvoke.org/
  295. targets R package https://cran.r-project.org/package=targets
  296. Compi https://doi.org/10.7717/peerj-cs.593 https://github.com/sing-group/compi https://www.sing-group.org/compi
  297. TriggerFlow: Event-based Orchestration of Serverless Workflows (https://github.com/triggerflow/triggerflow)
  298. Google Cloud Workflows: Orchestrate and automate Google Cloud and HTTP-based API services with serverless workflows. (https://cloud.google.com/workflows/docs)
  299. PanDA Workflow Management System: https://doi.org/10.1051/epjconf/201921403050 https://doi.org/10.1051/epjconf/201610801003 https://github.com/PanDAWMS
  300. Harvester: https://github.com/HSF/harvester
  301. BD-Processor https://github.com/big-data-processor/bd-processor/
  302. redun (yet another redundant workflow engine) https://github.com/insitro/redun
  303. pyiron (The materials science IDE) https://pyiron.org/