Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up unused triggers in a single query for all dialects except MySQL #38663

Merged
merged 1 commit into from Apr 2, 2024

Conversation

hussein-awala
Copy link
Member

This PR improves the performance of clean_unsed triggers method by executing the cleanup for Postgres and SQLite in a single query, which could reduce the latency and memory usage.

For MySQL, it doesn't support DELETE with JOIN, so we need to keep the current behavior:

[SQL: DELETE FROM `trigger` WHERE `trigger`.id IN (SELECT `trigger`.id 
FROM `trigger` LEFT OUTER JOIN task_instance ON `trigger`.id = task_instance.trigger_id GROUP BY `trigger`.id 
HAVING count(task_instance.trigger_id) = %s)]
[parameters: (0,)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

@hussein-awala hussein-awala added the type:improvement Changelog: Improvements label Apr 1, 2024
@hussein-awala hussein-awala added this to the Airflow 2.9.0 milestone Apr 1, 2024
@eladkal eladkal merged commit 12d38e1 into apache:main Apr 2, 2024
41 checks passed
ephraimbuddy pushed a commit that referenced this pull request Apr 2, 2024
ephraimbuddy pushed a commit that referenced this pull request Apr 2, 2024
ephraimbuddy pushed a commit that referenced this pull request Apr 2, 2024
idantepper pushed a commit to idantepper/airflow that referenced this pull request Apr 3, 2024
idantepper pushed a commit to idantepper/airflow that referenced this pull request Apr 3, 2024
mathiaHT pushed a commit to mathiaHT/airflow that referenced this pull request Apr 4, 2024
utkarsharma2 pushed a commit to astronomer/airflow that referenced this pull request Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:Triggerer type:improvement Changelog: Improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants