Test that DagFileProcessor can operator against on a Serialized DAG #8739
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Looks like it works fine :) |
ashb
commented
May 7, 2020
ashb
force-pushed
the
can-schedule-on-s10n-dag
branch
2 times, most recently
from
81e6ae8
to
e29c35c
Compare
ashb
commented
May 15, 2020
ashb
force-pushed
the
can-schedule-on-s10n-dag
branch
from
b36b276
to
f3dbc38
Compare
|
Just the Travis/Kube tests failing now. And they have been doing that for a while now :( |
ashb
force-pushed
the
can-schedule-on-s10n-dag
branch
from
26110c3
to
06932e0
Compare
As part of the scheduler HA work we are going to want to separate the parsing from the scheduling, so this changes the tests to ensure that the important methods of DagFileProcessor can do everything the need to when given a SerializedDAG, not just a DAG. i.e. that we have correctly serialized all the necessary fields.
ashb
force-pushed
the
can-schedule-on-s10n-dag
branch
from
06932e0
to
da07181
Compare
kaxil
approved these changes
May 29, 2020
dimberman
approved these changes
May 29, 2020
kaxil
reviewed
May 29, 2020
| @@ -117,7 +116,6 @@ def test_remove_stale_dags(self): | |||
| self.assertFalse(SDM.has_dag(stale_dag.dag_id)) | |||
| self.assertTrue(SDM.has_dag(fresh_dag.dag_id)) | |||
|
|
|||
| @mock.patch('airflow.models.serialized_dag.STORE_SERIALIZED_DAGS', True) | |||
| def test_bulk_sync_to_db(self): | |||
There was a problem hiding this comment.
How does this test work now, this doesn't store serialized DAG anymore, does it ?
There was a problem hiding this comment.
I mean with the patch, it would only sync and write DAGs to DAGModel table and assert that queries count is 7, is that the intention
There was a problem hiding this comment.
SDM.bulk_sync_to_db will now always write to the DB when called.
Before it would return if the config was false. Now the check is moved in to DagBag.bulk_sync:
if self.store_serialized_dags:
SerializedDagModel.bulk_sync_to_db(self.dags.values())There was a problem hiding this comment.
(This test isn't great, but that wasn't changed by this)
kaxil
approved these changes
May 29, 2020
turbaszek
approved these changes
May 30, 2020
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As part of the scheduler HA work we are going to want to separate the
parsing from the scheduling, so this changes the tests to ensure that
the important methods of DagFileProcessor can do everything the need to
when given a SerializedDAG, not just a DAG. i.e. that we have correctly
serialized all the necessary fields.
Make sure to mark the boxes below before creating PR: [x]
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.