You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
last week I had to start a lot of DAGs with catchup=True starting 01.01.2021 each 4 hours.
This DAG pulls Data from a Database, stores it in another Database and then migrates the Data into some target tables. In between I have some Data Quality checks.
When quality is not OK, the Dag Run fails and the "buffered" data is kept for reprocessing after the errors are solved.
I was quite uncomfortable with letting the DAGs run over night because I didn't want to come back next morning and finding out that there are thousands of DAG run fails.
One thing that would make me more comfortable is an option where I can tell a DAG if there are x subsequent DAG run fails, it should be paused. I could come back next morning, check the errors, solve it and make the DAG active again.
What do you think about this? Did you encounter the same situation? How do you solve such "unpleasant" feelings?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hi Airflow Community!
last week I had to start a lot of DAGs with catchup=True starting 01.01.2021 each 4 hours.
This DAG pulls Data from a Database, stores it in another Database and then migrates the Data into some target tables. In between I have some Data Quality checks.
When quality is not OK, the Dag Run fails and the "buffered" data is kept for reprocessing after the errors are solved.
I was quite uncomfortable with letting the DAGs run over night because I didn't want to come back next morning and finding out that there are thousands of DAG run fails.
One thing that would make me more comfortable is an option where I can tell a DAG if there are x subsequent DAG run fails, it should be paused. I could come back next morning, check the errors, solve it and make the DAG active again.
What do you think about this? Did you encounter the same situation? How do you solve such "unpleasant" feelings?
Beta Was this translation helpful? Give feedback.
All reactions