Design DTT1 - Define (DAG Directed Acyclic Graph)

[fcaffieri]

EPIC: #4495

Description

The objective of the issue is to define the directed graph orchestrator without cycles, to orchestrate the execution of the DTT modules.

It is necessary to analyze the following possible orchestrators to use and analyze the advantages and disadvantages of each one to make an appropriate decision.

Platforms

• https://github.com/PrefectHQ/prefect 🟢
• https://airflow.apache.org/docs/apache-airflow/stable/index.html 🟢
• https://docs.dagster.io/getting-started/hello-dagster 🟢
• https://www.dask.org/ 🟡 too close to data pipelines
• https://luigi.readthedocs.io/en/stable/ 🟡 config as code
• https://www.mage.ai/ 🟢
• https://github.com/temporalio/temporal 🟢

Python frameworks

• https://yadage.readthedocs.io/en/latest/ 🟡 not Widely adopted tool
• https://github.com/pygraphkit/graphtik 🔴 seems to does not provide ootb scheduler
• https://github.com/openstack/taskflow 🟢 -> PICK!
• https://pydoit.org/ 🟡 not powerful. Complex to understand. No dag
• https://github.com/Miksus/rocketry 🔴 complex dependency handling. No dag
• https://joblib.readthedocs.io/en/stable/ 🔴 no dag, explicit parallelism

Others

• https://github.com/dagu-dev/dagu?tab=readme-ov-file 💛 written in GO
• https://www.cypress.io/ 🔴 testing framework. written in JS
• https://testproject.io/ 🔴 just another test framework
• http://galenframework.com/ 🔴 just another test framework
• https://webdriver.io/ 🔴 just another test framework
• https://robotframework.org/ 🔴 Not a DAG scheduler. Just a human-readable testing framework

Restrictions on tool choice

• Python is desirable
• A single tool should cover what we need
• As long as it covers what we need, it should be as simple as possible.
• Flexibility/modularization
• Widely adopted tool - Maturity

Tool selection new criteria

Based on a discussion with @rauldpm and @jnasselle , we've decided to use a framework rather than a platform. This is motivated to avoid managing one extra service.

DoD

• Search as much as we can frameworks that could help us to achieve our goal. HINT: do not go deeper, just list it.
• Based on the search result, make a first cleanup based on the most important selection criteria (TBD)
• Based on a curated search result, go deeper and aim to have a max of three alternatives
• Pick one of those alternatives -> Taskflow

[jnasselle]

Update

Nice article about DAG as a formal model of a workflow https://www.prefect.io/blog/you-probably-dont-need-a-dag

Not aiming to change our current approach, but good reading

Define terminology

Based on industry/similar software, most of them use the next set of names

• task/operators: atomic and indivisible action
• workflow/flow: A workflow is a concatenation and combination of tasks/operator
• execution/job: a workflow/flow instance

[fcaffieri]

Update

Install Airflow DAG to perform a basic proof of concept.

At the moment it seems that it has too many functionalities that we will not use, it looks very powerful.

Something important to highlight is that at first glance, the creation of taskflows must be done using Python code.

Continue the analysis of Airflow and comparative between the others.

---

Restrictions

Python is desirable 🟢
A single tool should cover what we need 🟢
As long as it covers what we need, it should be as simple as possible. 🟡
Flexibility/modularization 🟢
Widely adopted tool - Maturity 🟢

[jnasselle]

Update - Testing Prefect

Restrictions

Python is desirable 🟢
A single tool should cover what we need 🟡
As long as it covers what we need, it should be as simple as possible. 🟡
Flexibility/modularization 🟢
Widely adopted tool - Maturity 🟢

[jnasselle]

Update - Testing Dagster

Python is desirable 🟢
A single tool should cover what we need 🟡
As long as it covers what we need, it should be as simple as possible. 🟢
Flexibility/modularization 🟢
Widely adopted tool - Maturity 🟢

Not going further given the new design criteria: from platforms to frameworks

[fcaffieri]

Update - Testing Yadage

Yadage use JSON Reference to develop the workflows

---

Restrictions

Python is desirable 🟢
A single tool should cover what we need 🟡
As long as it covers what we need, it should be as simple as possible. 🟡
Flexibility/modularization
Widely adopted tool - Maturity

---

Implementation

• Easy to install.
• Not having a front-end to visualise the workflow.

[jnasselle]

Update - Testing Taskflow

A library to do [jobs, tasks, flows] in a highly available, easy to understand and declarative manner (and more!) to be used with OpenStack and other projects.

https://wiki.openstack.org/wiki/TaskFlow/Paradigm_shifts

Restrictions

Python is desirable 🟢
A single tool should cover what we need 🟡 no yaml as dag input
As long as it covers what we need, it should be as simple as possible. 🟢
Flexibility/modularization . 🟢
Widely adopted tool - Maturity . 🟢

---

Implementation

• Examples show the flexibility and how powerful it can be https://github.com/openstack/taskflow/tree/stable/2023.2/taskflow/examples
• Maturity: is used by other Openstack tools, meaning that maintenance is almost assured
• Not only DAG, but other kind of flows: linear, unordered
  

[fcaffieri]

Update

Finish the schema validation for the input yaml used by the task flow.
The next step is to incorporate this validator into the task flow.

Input yaml example:
inputTask.yml.txt

Schema definition:
schema.json.txt

Validator:
schemeValidator_v2.py.txt

[jnasselle]

Update

Based on previous investigations about DAG engines and the results that give TaskFlow the pick, we agreed to attempt our DAG engine with the bare basic functionality based on the next criteria

• No framework provides OOTB functionality: a lot of glue is needed (YAML parser, workflow to TaskFlow,....)
• No framework has outstanding maintenance or is widely adopted
• Python TopologicalSort (graphlib) is now part of the standard

Conclusion

We are going to develop our DAG engine based on graphlib

[QU3B1M]

LGTM!

[fcaffieri]

LGTM