You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Since whitebox workflows become more and more complex, we need a way to orchestrate them. We can use the Airflow Python API to define and execute workflows. A workflow is defined as a directed acyclic graph (DAG) in Airflow. Each node in the DAG represents a task, and the edges between nodes represent dependencies between tasks.
The architecture should become roughly as follows:
sequenceDiagram
participant API as API
participant Database as Database
participant Airflow as Airflow
API ->> Database: Store data
loop Cron Workflows
Database ->> Airflow: Extract data
Airflow ->> Airflow: Analyze data
Airflow ->> Database: Store result
end
Some implementation notes:
Airflow should be a different deployment that the API. Approach it as a different service that runs all the workflows.
It should use the same instance of SQL but have its own database as a database backend
All current (and future) metrics calculations should happen inside airflow.
The text was updated successfully, but these errors were encountered:
To start development, You can use this compose and dockerfile that I made for some local tests.
Just place your dags in a dags folder in the root of the projects (or change the path in the volumes of webserver to ./whitebox/dags:/usr/local/airflow/dags)
Dockerfile:
FROM puckel/docker-airflow:1.10.9
RUN pip install evidently==0.2.0
Since whitebox workflows become more and more complex, we need a way to orchestrate them. We can use the Airflow Python API to define and execute workflows. A workflow is defined as a directed acyclic graph (DAG) in Airflow. Each node in the DAG represents a task, and the edges between nodes represent dependencies between tasks.
The architecture should become roughly as follows:
Some implementation notes:
The text was updated successfully, but these errors were encountered: