Congestion Network 2.0: Automate Data Aggregation

[chmnata]

No description provided.

[chmnata]

Automate DAG needs to run the following functions

• generate_network_daily
• generate_network_monthly (aggregate from daily)
• generate_network_weekly (aggregate from daily)

So the DAG will use an external sensor to wait for pull_here, then runs the generate daily function, after that, check for the current date, and check if all the necessary dates are in the daily table, if so run the monthly and weekly functions on a weekly and monthly basis (or if we have airflow 2 use timetables).

[chmnata]

Final DAG flow:
Wait for here -> aggregate daily values -> Check day of month -> If its the first day of the month, check to see if the previous month's data are all available -> aggregate monthly values