You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
I added aql.cleanup() to a dag to clear temp tables.
it will run after all task success. however, if there are failures in the tasks. it will hang, with the
following errors:
[2022-07-12, 09:19:35 UTC] {cleanup.py:103} INFO - waiting on the following tasks to complete before cleaning up: [('complete', 'upstream_failed'), ('tear_down_tasks', 'removed'), ('create_staging_comments', 'upstream_failed'), ('merge_comments', 'upstream_failed'), ('upload_comments_to_gcs', 'upstream_failed'), ('extract_finish', 'upstream_failed'), ('create_comments', 'upstream_failed'), ('load_comments_to_snowflake_staging_db', 'upstream_failed'), ('cleanup_staging_comments', 'upstream_failed')]
[2022-07-12, 09:19:35 UTC] {cleanup.py:137} WARNING - You are currently running the 'waiting mode', where the task will wait for all other tasks to complete. Please note that for asynchronous executors (e.g. SequentialExecutor and DebugExecutor, this mode will cause locks.
Version
Astro: [e.g. 0.11.0]
OS: [eg. Debian]
Expected behavior
Cleanup would run whether all tasks success or if there are any task fail.
The text was updated successfully, but these errors were encountered:
guohui-gao
changed the title
aql.cleanup() will only clean up after all tasks success,
aql.cleanup() will hang if there are failed tasks.
Jul 12, 2022
Describe the bug
I added aql.cleanup() to a dag to clear temp tables.
it will run after all task success. however, if there are failures in the tasks. it will hang, with the
following errors:
[2022-07-12, 09:19:35 UTC] {cleanup.py:103} INFO - waiting on the following tasks to complete before cleaning up: [('complete', 'upstream_failed'), ('tear_down_tasks', 'removed'), ('create_staging_comments', 'upstream_failed'), ('merge_comments', 'upstream_failed'), ('upload_comments_to_gcs', 'upstream_failed'), ('extract_finish', 'upstream_failed'), ('create_comments', 'upstream_failed'), ('load_comments_to_snowflake_staging_db', 'upstream_failed'), ('cleanup_staging_comments', 'upstream_failed')]
[2022-07-12, 09:19:35 UTC] {cleanup.py:137} WARNING - You are currently running the 'waiting mode', where the task will wait for all other tasks to complete. Please note that for asynchronous executors (e.g. SequentialExecutor and DebugExecutor, this mode will cause locks.
Version
Expected behavior
Cleanup would run whether all tasks success or if there are any task fail.
The text was updated successfully, but these errors were encountered: