Closed
Conversation
boring-cyborg
bot
added
area:dev-tools
area:Scheduler
anitakar
force-pushed
the
store_dags_cleanup_1_10
branch
2 times, most recently
from
8614c32 to
cf2929c
Compare
Added missing store_serialized_dags to the _DagBag class. Read store_serialized_dags from config This value was read once when Airflow (scheduler, worker, web server) was started. We want web server (which is long runnning process) to read this value afresh every time. Only access dags from db not disk in a webserver Scheduler stopped parsing dags because DagBag::collect_dags method that it uses, returned no dags, as it does not collect dags when stored_serialized_dags is on. Generally, even though store_serialized_dags parameter is used in models and throughout common code, it should only be set to True in web server. If it is set to true in other parts of the system, they will stop working. CLI will stop working. Scheduler is the element that creates serialized dags and it should always read from disk to keep other parts of the system up to date. But there is no way to check whether dag_bag was called by web server and store_serialized_dags should be applied or if it was called from scheduler and dags should be read from files on the disk not from the db.
anitakar
force-pushed
the
store_dags_cleanup_1_10
branch
from
cf2929c to
7f7dc0f
Compare
6 tasks
ashb
reviewed
May 7, 2020
ashb
requested changes
May 7, 2020
ashb
removed
area:Scheduler
Member
|
I'm going to close this, as the main feature here is not needed with Dag serialization on, and the other changes are a performance regression. We can re-open this is you disagree. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make sure to mark the boxes below before creating PR: [x]
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.