Shell supporting pipelines to and from multiple processes
Switch branches/tags
Clone or download
Latest commit 94c8184 Oct 20, 2018

The Directed Graph Shell (dgsh)

Build Status

The directed graph shell, dgsh, allows the expressive expression of efficient big data set and streams processing pipelines using existing Unix tools as well as custom-built components. It is a Unix-style shell allowing the specification of pipelines with non-linear scatter-gather operations. These form a directed acyclic process graph, which is typically executed by multiple processor cores, thus increasing the operation's processing throughput.

You can find a complete introduction, reference documentation, and illustrated examples in the suite's web site.

See also, a quick video overview and the associated (open access) paper, Extending Unix pipelines to DAGs, published in the IEEE Transactions on Computers, 66(9):1547–1561, 2017.