Add conditional logic for dataset triggering #37016
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dstandish
force-pushed
the
add-conditional-logic-for-dataset-triggering
branch
from
cd01fde
to
859a0ac
Compare
1 task
dstandish
force-pushed
the
add-conditional-logic-for-dataset-triggering
branch
2 times, most recently
from
0e1e5d8
to
ecccd6f
Compare
update: the performance concern appears to be invalid, due to randomness in task execution issues / unrelated scheduler restarts |
sunank200
force-pushed
the
add-conditional-logic-for-dataset-triggering
branch
5 times, most recently
from
83087fc
to
3cffcf4
Compare
sunank200
force-pushed
the
add-conditional-logic-for-dataset-triggering
branch
3 times, most recently
from
7514d44
to
ab979da
Compare
Documentation changes are done here |
2 tasks
dstandish
changed the title
dstandish
requested review from
uranusjr,
kaxil,
ashb,
bolkedebruin and
XD-DENG
as code owners
uranusjr
reviewed
Feb 20, 2024
uranusjr
reviewed
Feb 20, 2024
dstandish
force-pushed
the
add-conditional-logic-for-dataset-triggering
branch
from
870452c
to
9753973
Compare
This reverts commit 2a2a195.
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
dstandish
force-pushed
the
add-conditional-logic-for-dataset-triggering
branch
from
0137de6
to
71f6eba
Compare
Co-authored-by: Wei Lee <weilee.rx@gmail.com> Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Created issue on our board @bbovenzi |
jedcunningham
approved these changes
Feb 21, 2024
This was referenced Feb 22, 2024
abhishekbhakat
pushed a commit
to abhishekbhakat/my_airflow
that referenced
this pull request
Mar 5, 2024
Add conditional logic for dataset-triggered dags so that we can schedule based on dataset1 OR dataset1. This PR only implements the underlying classes, DatasetAny and DatasetAll. In a followup PR we will add more convenient syntax for this, specifically the | and & symbols, e.g. (dataset1 | dataset2) & dataset3. --------- Co-authored-by: Ankit Chaurasia <8670962+sunank200@users.noreply.github.com> Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add conditional logic for dataset-triggered dags.
This means we can schedule based on
dataset1 OR dataset1.This PR only implements the underlying classes,
DatasetAnyandDatasetAll. In a followup PR we will add more convenient syntax for this, specifically the|and&symbols, e.g.(dataset1 | dataset2) & dataset3.