Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Openlineage support - Add Extractor for MergeOperator #900

Closed
8 tasks
kaxil opened this issue Sep 23, 2022 · 0 comments · Fixed by #1068
Closed
8 tasks

Openlineage support - Add Extractor for MergeOperator #900

kaxil opened this issue Sep 23, 2022 · 0 comments · Fixed by #1068
Assignees
@sunank200
Labels
feature New feature or request priority/high High priority product/python-sdk Label describing products
Milestone
1.2.0

Comments

@kaxil
Copy link
Collaborator

kaxil commented Sep 23, 2022 •
edited
Loading

Please describe the feature you'd like to see
We should be able to extract open lineage info from the MergeOperator.

Describe the solution you'd like

Acceptance Criteria

  • Post the screenshot of how it looks in the Openlineage/Marquez UI
  • All checks and tests in the CI should pass
  • Unit tests (90% code coverage or more, once available)
  • Integration tests (if the feature relates to a new database or external service)
  • Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
  • Exception handling in case of errors
  • Logging (are we exposing useful information to the user? e.g. source and destination)
  • Improve the documentation (README, Sphinx, and any other relevant)
@kaxil kaxil added priority/high High priority product/python-sdk Label describing products feature New feature or request labels Sep 23, 2022
@kaxil kaxil added this to the 1.2.0 milestone Sep 23, 2022
@kaxil kaxil mentioned this issue Sep 23, 2022
Closed
13 tasks
@sunank200 sunank200 assigned sunank200 and unassigned utkarsharma2 Oct 14, 2022
@sunank200 sunank200 mentioned this issue Oct 14, 2022
Merged
2 tasks
sunank200 added a commit that referenced this issue Oct 17, 2022
@sunank200 @pre-commit-ci
Add open lineage support for MergeOperator (#1068)
454fbf9 
# Description
## What is the current behavior?
<!-- Please describe the current behavior that you are modifying. -->
Open Lineage support for `MergeOperator` is missing.

<!--
Issues are required for both bug fixes and features.
Reference it using one of the following:

closes: #ISSUE
related: #ISSUE
-->
closes: #900 


## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->
- Add open Lineage support for `MergeOperator`
- Add the test for the same

## Does this introduce a breaking change?
No

### Checklist
- [x] Created tests which fail without the change (if possible)
- [x] Extended the README / documentation, if necessary

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sunank200 sunank200

Labels
feature New feature or request priority/high High priority product/python-sdk Label describing products
Projects
None yet
Milestone
1.2.0
Development

Successfully merging a pull request may close this issue.

Add open lineage support for MergeOperator astronomer/astro-sdk
3 participants
@sunank200 @kaxil @utkarsharma2