Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aql.cleanup() will hang if there are failed tasks. #525

Closed
guohui-gao opened this issue Jul 12, 2022 · 0 comments · Fixed by #540
Closed

aql.cleanup() will hang if there are failed tasks. #525

guohui-gao opened this issue Jul 12, 2022 · 0 comments · Fixed by #540
Labels
bug Something isn't working priority/high High priority
Milestone

Comments

@guohui-gao
Copy link

Describe the bug
I added aql.cleanup() to a dag to clear temp tables.
it will run after all task success. however, if there are failures in the tasks. it will hang, with the
following errors:

[2022-07-12, 09:19:35 UTC] {cleanup.py:103} INFO - waiting on the following tasks to complete before cleaning up: [('complete', 'upstream_failed'), ('tear_down_tasks', 'removed'), ('create_staging_comments', 'upstream_failed'), ('merge_comments', 'upstream_failed'), ('upload_comments_to_gcs', 'upstream_failed'), ('extract_finish', 'upstream_failed'), ('create_comments', 'upstream_failed'), ('load_comments_to_snowflake_staging_db', 'upstream_failed'), ('cleanup_staging_comments', 'upstream_failed')]
[2022-07-12, 09:19:35 UTC] {cleanup.py:137} WARNING - You are currently running the 'waiting mode', where the task will wait for all other tasks to complete. Please note that for asynchronous executors (e.g. SequentialExecutor and DebugExecutor, this mode will cause locks.

Version

  • Astro: [e.g. 0.11.0]
  • OS: [eg. Debian]

Expected behavior
Cleanup would run whether all tasks success or if there are any task fail.

@guohui-gao guohui-gao changed the title aql.cleanup() will only clean up after all tasks success, aql.cleanup() will hang if there are failed tasks. Jul 12, 2022
@kaxil kaxil added the bug Something isn't working label Jul 12, 2022
@kaxil kaxil added this to the 1.0.0 milestone Jul 12, 2022
@kaxil kaxil added the priority/high High priority label Jul 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working priority/high High priority
Projects
None yet
2 participants