DTT1 - Iteration 2 - QA Workflow Engine

[jnasselle]

Epic: #4495

---

Description

This issue aims to develop the bare basic implementation of a DAG engine

Tasks

• Define a YAML input file schema and mechanism to handle DAG definition that allows templates to improve design logic.
  • Use DTT1 agent / DTT1 agent as use cases for such definition
  • Provide a YAML schema validator to decouple the validation
  • Provide a bunch of YAML examples, from simple to complex
• Allow dry-run execution
• Allow parallel execution and explicitly set how many threads will be used
• Task could execute some pre-agreed tasks that will be executed with its own parameters logic
  • process: allow the execution of a process
  • dummy
  • dummy-random
• requirements.txt
• README.md
• Unit tests

Known issues

• Task duplication: there's no validation that two tasks have the same name after expansion
• Missing task validation: dry-run does not validate task existence. It fails during runtime
• cleanup feature (code support): allow optional cleanup block in tasks, that allows undo dag execution to release resources or whatever scenario
• on-error feature: allow pre-agreed handling mechanism if a task fails
  • continue
  • abort-related-flows -> This should be the default value
  • abort-all

[fcaffieri]

Update

• Added the schema validations to the Workflow engine.
• Fix the requirements.txt adding all the new library dependencies.
• Add schema.json with the validator template.
• Add new class SchemaValidator to handle the validations.
• Execute test with the Workflow Engine.

[QU3B1M]

LGTM!

[fcaffieri]

Unit test will be launched in #4751 Integrating this new module with the others.
LGTM