Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve trigger assign_unassigned method #38664

Merged
merged 1 commit into from Apr 2, 2024

Conversation

hussein-awala
Copy link
Member

Improve trigger assign_unassigned method by merging alive_triggerer_ids and get_sorted_triggers queries to reduce latency and memory usage.

@hussein-awala hussein-awala added this to the Airflow 2.9.0 milestone Apr 2, 2024
@hussein-awala hussein-awala added the type:improvement Changelog: Improvements label Apr 2, 2024
@uranusjr
Copy link
Member

uranusjr commented Apr 2, 2024

Do we even need to do this outside of get_sorted_triggers in the first place? I wonder why we can’t just pass health_check_threshold into the function and do the subquery there instead…

@hussein-awala
Copy link
Member Author

Yes, we can do that, but I think keeping it like this will keep the unit tests simple, as we can pass the IDs in a list instead of setting up the trigger table.

@uranusjr
Copy link
Member

uranusjr commented Apr 2, 2024

Maybe let’s add some annotation or docstring on get_sorted_triggers so future people know what to expect? At least they need to know to expect a subquery, not always a Python list or some iterable.

@hussein-awala hussein-awala merged commit 8246a89 into apache:main Apr 2, 2024
39 of 41 checks passed
ephraimbuddy pushed a commit that referenced this pull request Apr 2, 2024
…get_sorted_triggers queries (#38664)

(cherry picked from commit 8246a89)
ephraimbuddy pushed a commit that referenced this pull request Apr 2, 2024
…get_sorted_triggers queries (#38664)

(cherry picked from commit 8246a89)
idantepper pushed a commit to idantepper/airflow that referenced this pull request Apr 3, 2024
idantepper pushed a commit to idantepper/airflow that referenced this pull request Apr 3, 2024
mathiaHT pushed a commit to mathiaHT/airflow that referenced this pull request Apr 4, 2024
utkarsharma2 pushed a commit to astronomer/airflow that referenced this pull request Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:Triggerer type:improvement Changelog: Improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants