Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't show stale Serialized DAGs if they are deleted in DB #16368

Merged
merged 1 commit into from Jun 11, 2021

Conversation

@kaxil
Copy link
Member

@kaxil kaxil commented Jun 10, 2021

If DagBag.get_dag() is called currently, it will return the DAG
even if the DAG does not exist in serialized_dag table.

This PR changes that logic to remove the dag from local cache too
when DagBag.get_dag() is called. This happens after
min_serialized_dag_fetch_secs.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@kaxil kaxil force-pushed the dags-remove-dagbag branch from a8c7c73 to b533794 Jun 10, 2021
@kaxil kaxil requested a review from ephraimbuddy Jun 10, 2021
@kaxil kaxil marked this pull request as ready for review Jun 10, 2021
@kaxil kaxil requested review from ashb, turbaszek and XD-DENG as code owners Jun 10, 2021
@kaxil kaxil force-pushed the dags-remove-dagbag branch from b533794 to a57c755 Jun 10, 2021
airflow/models/dagbag.py Outdated Show resolved Hide resolved
@kaxil kaxil force-pushed the dags-remove-dagbag branch from af4f6d1 to 42afb0d Jun 11, 2021
airflow/models/dagbag.py Outdated Show resolved Hide resolved
If `DagBag.get_dag()` is called currently, it will return the DAG
even if the DAG does not exist in `serialized_dag` table.

This PR changes that logic to remove the dag from local cache too
when `DagBag.get_dag()` is called. This happens after
`min_serialized_dag_fetch_secs`.
@kaxil kaxil force-pushed the dags-remove-dagbag branch from 42afb0d to fd42751 Jun 11, 2021
ashb
ashb approved these changes Jun 11, 2021
@github-actions
Copy link

@github-actions github-actions bot commented Jun 11, 2021

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@kaxil kaxil closed this Jun 11, 2021
@kaxil kaxil reopened this Jun 11, 2021
@kaxil kaxil merged commit e3b3c1f into apache:main Jun 11, 2021
71 of 77 checks passed
@kaxil kaxil deleted the dags-remove-dagbag branch Jun 11, 2021
@ashb ashb added this to the Airflow 2.1.1 milestone Jun 16, 2021
jhtimmins added a commit to astronomer/airflow that referenced this issue Jun 16, 2021
)

If `DagBag.get_dag()` is called currently, it will return the DAG
even if the DAG does not exist in `serialized_dag` table.

This PR changes that logic to remove the dag from local cache too
when `DagBag.get_dag()` is called. This happens after
`min_serialized_dag_fetch_secs`.

(cherry picked from commit e3b3c1f)
ashb added a commit that referenced this issue Jun 22, 2021
If `DagBag.get_dag()` is called currently, it will return the DAG
even if the DAG does not exist in `serialized_dag` table.

This PR changes that logic to remove the dag from local cache too
when `DagBag.get_dag()` is called. This happens after
`min_serialized_dag_fetch_secs`.

(cherry picked from commit e3b3c1f)
kaxil added a commit to astronomer/airflow that referenced this issue Jun 22, 2021
)

If `DagBag.get_dag()` is called currently, it will return the DAG
even if the DAG does not exist in `serialized_dag` table.

This PR changes that logic to remove the dag from local cache too
when `DagBag.get_dag()` is called. This happens after
`min_serialized_dag_fetch_secs`.

(cherry picked from commit e3b3c1f)
kaxil added a commit to astronomer/airflow that referenced this issue Jun 23, 2021
)

If `DagBag.get_dag()` is called currently, it will return the DAG
even if the DAG does not exist in `serialized_dag` table.

This PR changes that logic to remove the dag from local cache too
when `DagBag.get_dag()` is called. This happens after
`min_serialized_dag_fetch_secs`.

(cherry picked from commit e3b3c1f)
(cherry picked from commit 9d14b1d)
kaxil added a commit to astronomer/airflow that referenced this issue Jun 23, 2021
)

If `DagBag.get_dag()` is called currently, it will return the DAG
even if the DAG does not exist in `serialized_dag` table.

This PR changes that logic to remove the dag from local cache too
when `DagBag.get_dag()` is called. This happens after
`min_serialized_dag_fetch_secs`.

(cherry picked from commit e3b3c1f)
(cherry picked from commit 9d14b1d)
(cherry picked from commit c95f6d9)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants