Placeholder for shell tasks which expand into inner DAGs

[alexanderdean]

This functionality is in support of:

• EmrEtlRunner: placeholder for "Factotum inside"
• Placeholder for "Factotum inside" for SQL Runner

Pre the above tickets, some command-line tools will provide a command-line option to yield an inner Factotum DAG that can then be run from inside a parent Factotum DAG. This is a key plank of rule 2 of the Zen of Factotum:

A job must be composable from other jobs

Current thinking (feedback welcome) is as follows:

• Add a new DAG "task"(?) type - this needs further discussion as Factotum 0.1.0 just assumes that all tasks are tasks (not inner DAGs)
• Add support for the DAG task type to be populated at runtime using a shell command (we should futureproof the schema design so that we can also support the simpler use case where an inner DAG is held in a file, see Add support for a Factotum factfile as an inner DAG #40, or KV-store)
• When Factotum starts to run the job, it will evaluate the shell command which will yield the inner DAG in Factotum DAG format via stdout and return 0
• Factotum will validate the inner DAG to ensure that it's a valid Factotum DAG
• Factotum will then expand the parent DAG to include the inner DAG at the correct point

In the integration tests for this feature, we should write a shell script which returns a simple Factotum DAG into a parent DAG, to validate that this is working correctly.

See also #40 for the sibling ticket where the simpler case of an inner DAG being retrievable from a static file is considered.

[alexanderdean]

An alternative would be to have a converter built into Factotum which can convert e.g. SQL Runner playbooks to Factotum DAGs.