Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the dataset_expression as part of DagModel and DAGDetailSchema #37826

Merged
merged 2 commits into from Mar 8, 2024

Conversation

sunank200
Copy link
Collaborator

@sunank200 sunank200 commented Mar 1, 2024

  • Add the dataset_expression as part of DagModel and DAGDetailSchema

For example, the following dataset expression: (dag1_dataset | (dag2_dataset & dag3_dataset)) is returned as part of dictionary.

"dataset_expression": {
    "any": [
      "s3://dag1/output_1.txt",
      {
        "all": [
          "s3://dag2/output_1.txt",
          "s3://dag3/output_3.txt"
        ]
      }
    ]
  }

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:serialization labels Mar 1, 2024
@sunank200 sunank200 force-pushed the update_get_dag_details branch 2 times, most recently from 5f1f577 to 55e230f Compare March 1, 2024 16:40
@sunank200 sunank200 changed the title Add the dataset_expression as part of get_dag_details Add the dataset_expression as part of DAGDetailSchema Mar 1, 2024
@sunank200 sunank200 marked this pull request as ready for review March 1, 2024 20:12
@uranusjr
Copy link
Member

uranusjr commented Mar 2, 2024

I kind of wonder if this should cover the timetable as well i.e. have schedule_expression instead of special-casing datasets. We can maybe also put this on DagModel directly.

@sunank200
Copy link
Collaborator Author

@uranusjr as per conversation on slack I have done the following change:

But I feel this should not be done in the schema at all, instead this JSON should be generated when a DagModel is created (before serialization happens), and the schema should just need to read the field out

airflow/models/dag.py Outdated Show resolved Hide resolved
airflow/models/dag.py Outdated Show resolved Hide resolved
airflow/models/dag.py Outdated Show resolved Hide resolved
@sunank200 sunank200 force-pushed the update_get_dag_details branch 3 times, most recently from bdc0f61 to 28cddb0 Compare March 7, 2024 09:42
@sunank200 sunank200 changed the title Add the dataset_expression as part of DAGDetailSchema Add the dataset_expression as part of DagModel and DAGDetailSchema Mar 7, 2024
airflow/models/dag.py Outdated Show resolved Hide resolved
@sunank200 sunank200 requested a review from potiuk as a code owner March 7, 2024 14:42
@sunank200 sunank200 force-pushed the update_get_dag_details branch 4 times, most recently from d19aa74 to a2e1101 Compare March 7, 2024 15:50
@sunank200 sunank200 requested a review from uranusjr March 7, 2024 15:50
@sunank200 sunank200 requested a review from uranusjr March 8, 2024 06:19
airflow/models/dag.py Outdated Show resolved Hide resolved
Add the dataset_expression as part of DAGDetailSchema.

Add dataset_expression in DAGDetailSchema

Use dataset_triggers on the DAG to create dataset_expression on DagModel

Simplify

Fix dataset_expression default

Fix the test and add database migration

Add tests for dataset_expression
airflow/models/dag.py Show resolved Hide resolved
@phanikumv phanikumv merged commit 832d2f5 into apache:main Mar 8, 2024
56 checks passed
@phanikumv phanikumv deleted the update_get_dag_details branch March 8, 2024 10:40
@eladkal eladkal added this to the Airflow 2.9.0 milestone Mar 8, 2024
@ephraimbuddy ephraimbuddy added the type:new-feature Changelog: New Features label Mar 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:API Airflow's REST/HTTP API area:serialization type:new-feature Changelog: New Features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants