-
Notifications
You must be signed in to change notification settings - Fork 40
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Openlineage support - Add Extractor for
DataframeOperator
(#1183)
# Description ## What is the current behavior? We are unable to extract open lineage info from the DataframeOperator. closes: #904 ## What is the new behavior? <!-- Please describe the behavior or changes that are being added by this PR. --> - Added a method get_openlineage_facets on the Operator (based on [this doc](https://docs.google.com/document/d/1vPsvHejQ24xTbzpz_LYSf0_ixk9oUuBiEUHVEaF9J2U/edit?usp=sharing)) - The "PythonSDKExtractor" built-in #898 should be able to work with DataframeOperator.get_openlineage_facets. Tested it to make sure it works ## Does this introduce a breaking change? No ### Checklist - [x] Post the screenshot of how it looks in the Openlineage/Marquez UI - [x] All checks and tests in the CI should pass - [x] Unit tests (90% code coverage or more, #191) - [ ] Integration tests (if the feature relates to a new database or external service) - [ ] Docstrings in [reStructuredText](https://peps.python.org/pep-0287/) for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used) - [ ] Exception handling in case of errors - [ ] Logging (are we exposing useful information to the user? e.g. source and destination) - [ ] Improve the documentation (README, Sphinx, and any other relevant) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com> Co-authored-by: Pankaj <pankaj.singh@astronomer.io>
- Loading branch information
1 parent
2b4ff04
commit 4785827
Showing
3 changed files
with
105 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters