Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Listener hooks for Datasets #34418

Merged
merged 17 commits into from Nov 13, 2023
Merged

Conversation

yermalov-here
Copy link
Contributor

This PR creates listener hooks for the following Dataset events

  • on_dataset_created
  • on_dataset_changed

closes: #34327


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@hussein-awala hussein-awala added type:new-feature Changelog: New Features area:datasets Issues related to the datasets feature labels Sep 16, 2023
@hussein-awala hussein-awala added this to the Airflow 2.8.0 milestone Sep 16, 2023
@potiuk
Copy link
Member

potiuk commented Nov 3, 2023

Some tests shoudl be marked as DB tests- see the error and link describing what needs to be done.

@yermalov-here
Copy link
Contributor Author

Some tests shoudl be marked as DB tests- see the error and link describing what needs to be done.

got it. will take a look

@mobuchowski mobuchowski self-requested a review November 6, 2023 15:26
@mobuchowski
Copy link
Contributor

One question - on which Airflow component (scheduler, worker...) datasets are updated?

It would be nice to add this to docstring, probably for the rest of the methods (in some separate PR). For now it's only mentioned here.

@yermalov-here
Copy link
Contributor Author

One question - on which Airflow component (scheduler, worker...) datasets are updated?

It would be nice to add this to docstring, probably for the rest of the methods (in some separate PR). For now it's only mentioned here.

i can be wrong due to the lack of knowledge, but afaiu scheduler is responsible for creating new datasets and workers for updating them.

@kaxil kaxil merged commit 9439111 into apache:main Nov 13, 2023
45 of 46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:datasets Issues related to the datasets feature kind:documentation type:new-feature Changelog: New Features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Listeners for Datasets
7 participants