Skip to content

Comments

Reduce round trips to db when evaluating setup tasks#34578

Closed
dstandish wants to merge 1 commit intoapache:mainfrom
astronomer:reduce-round-trips-to-db-for-trigger-rule-dep
Closed

Reduce round trips to db when evaluating setup tasks#34578
dstandish wants to merge 1 commit intoapache:mainfrom
astronomer:reduce-round-trips-to-db-for-trigger-rule-dep

Conversation

@dstandish
Copy link
Contributor

Ensuring that cleared tasks wait for indirect setup tasks was fixed in #33903. That fix entailed applying logic similar to ALL_SUCCESS logic for just the indirect upstream tasks before moving on to the evaluation of direct upstreams.

This had the benefit of similicity and isolation -- it would only be called for tasks that had indirect upstreams and it would be layered on top of the existing trigger rule dep logic -- not mixed in.

But it resulted in new round trips to the db when there is an indirect setup.

What this PR does is try to get all the upstream TI info in one query.

Ensuring that cleared tasks wait for indirect setup tasks was fixed in apache#33903.  That fix entailed applying logic similar to ALL_SUCCESS logic for just the indirect upstream tasks before moving on to the evaluation of direct upstreams.

This had the benefit of similicity and isolation -- it would only be called for tasks that had indirect upstreams and it would be layered on top of the existing trigger rule dep logic -- not mixed in.

But it resulted in new round trips to the db when there is an indirect setup.

What this PR does is try to get all the upstream TI info in one query.
@github-actions
Copy link

github-actions bot commented Nov 8, 2023

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Nov 8, 2023
@github-actions github-actions bot closed this Nov 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale Stale PRs per the .github/workflows/stale.yml policy file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant