Epic: Directed Acyclic Graph (DAG) pipelines

[kminehart]

Shipwright pipelines should store steps in a Directed Acyclic Graph rather than a queue.

Using a DAG would allow us to do what most CI platforms do with extensive branching.

Assuming each row of nodes is ran in parallel, right now, we can do this:

But without a DAG, we can not do this:

Tasks:

• Add a Sub pipeline interface that allows pipeline developers to create async branches in pipelines
• Add a DAG type and utility functions
• Create an example with a more highly branching execution tree
• Output a more complex pipeline in Drone.

Details:

I think we can do this with defining extensive branching pipelines without completely refactoring everything:

type SubFunc func(s *Shipwright)

func (s *Shipwright) Sub(sf SubFunc) {}

And in a pipeline (represents the above graph):

sw := shipwright.New("some pipeline")
defer sw.Done()

sw.Run(Step1(...))

sw.Parallel(Step2(...), Step3(...))

sw.Sub(func(sw *shipwright.Shipwright) {
    sw.Run(Step3a(...), Step3b(...))
})

sw.Run(Step4(...))

sw.Parallel(Step5(...), Step6(...), Step7(...))