Don't create session in get_dag if not reading dags from database #38553
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Tests failing. |
dstandish
force-pushed
the
dont-make-unnecessary-network-calls-with-dag-related-commands
branch
from
786b0c7
to
9cc89e6
Compare
should be resolved now 🤞 |
dstandish
force-pushed
the
dont-make-unnecessary-network-calls-with-dag-related-commands
branch
from
9cc89e6
to
b0c244c
Compare
The function DagBag.get_dag has some logic to check if the serialized dag object should be refreshed. By accessing the dags dict directly we avoid that logic, and this is helpful for running a task exclusively with the internal API. Generally speaking this function "get_dag" is invoked when running the task command on a worker, and in that context we shouldn't need to handle expiration of the serdag. For the case where we're reading from db, I have left the original method call in place (which keeps the expiration logic in effect for that scenario).
dstandish
force-pushed
the
dont-make-unnecessary-network-calls-with-dag-related-commands
branch
from
b0c244c
to
6a06ee4
Compare
dstandish
changed the title
uranusjr
reviewed
Apr 2, 2024
potiuk
approved these changes
Apr 2, 2024
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
dstandish
deleted the
dont-make-unnecessary-network-calls-with-dag-related-commands
branch
ephraimbuddy
added
type:improvement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The function DagBag.get_dag has some logic to check if the serialized dag object should be refreshed. By accessing the dags dict directly we avoid that logic, and this is helpful for running a task exclusively with the internal API. Generally speaking this function "get_dag" is invoked when running the task command on a worker, and in that context we shouldn't need to handle expiration of the serdag. For the case where we're reading from db, I have left the original method call in place (which keeps the expiration logic in effect for that scenario).