Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make openlineage an optional dependency #1252

Merged
merged 7 commits into from
Nov 16, 2022
Merged

Make openlineage an optional dependency #1252

merged 7 commits into from
Nov 16, 2022

Conversation

sunank200
Copy link
Contributor

Description

What is the current behavior?

We shouldn’t import the openlineage package at top level if it is optional.
Refer to https://astronomer.slack.com/archives/C02B8SPT93K/p1668186729410429

closes: #1243

What is the new behavior?

  • Add a try except for import error
  • Remove the open lineage from test in pyproject

Does this introduce a breaking change?

No

Checklist

  • Created tests which fail without the change (if possible)
  • Extended the README / documentation, if necessary

@codecov
Copy link

codecov bot commented Nov 15, 2022 •
edited
Loading

Codecov Report

Base: 94.13% // Head: 94.13% // No change to project coverage 👍

Coverage data is based on head (13e0ed4) compared to base (a99f949).
Patch has no changes to coverable lines.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main    #1252   +/-   ##
=======================================
  Coverage   94.13%   94.13%           
=======================================
  Files          17       17           
  Lines         597      597           
  Branches       67       67           
=======================================
  Hits          562      562           
  Misses         22       22           
  Partials       13       13

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

kaxil
kaxil reviewed Nov 15, 2022
View reviewed changes
python-sdk/noxfile.py Outdated Show resolved Hide resolved
kaxil
kaxil reviewed Nov 15, 2022
View reviewed changes
python-sdk/src/astro/sql/operators/append.py Outdated
Comment on lines 9 to 21
try:
from openlineage.client.facet import (
BaseFacet,
DataQualityMetricsInputDatasetFacet,
DataSourceDatasetFacet,
OutputStatisticsOutputDatasetFacet,
SchemaDatasetFacet,
SchemaField,
SqlJobFacet,
)
from openlineage.client.run import Dataset as OpenlineageDataset
except ImportError:
logging.warning("Install openlineage-airflow")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I don't like the try..except in all the modules, since it is used everywhere.. We can just try to import it once in some common module, and use that.

kaxil
kaxil reviewed Nov 15, 2022
View reviewed changes
python-sdk/src/astro/sql/operators/load_file.py Outdated
try:
from openlineage.client.facet import BaseFacet, DataSourceDatasetFacet, SchemaDatasetFacet, SchemaField
from openlineage.client.run import Dataset as OpenlineageDataset
except ImportError:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line 21 still will import openlineage library

@kaxil kaxil changed the title Import the openlineage package is optional. Make openlineage an optional dependency Nov 15, 2022
@kaxil
Copy link
Collaborator

kaxil commented Nov 15, 2022

I have pushed a different way of handling this in a896896

feluelle
feluelle reviewed Nov 15, 2022
View reviewed changes
python-sdk/src/astro/lineage/__init__.py Outdated
Comment on lines 20 to 22
has_openlineage = True
except ImportError:
has_openlineage = False
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have these? If we are not using them?

Copy link
Member

@feluelle feluelle Nov 15, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we not use has_openlineage in the other modules to decide if we want to import or not?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

Can we not use has_openlineage in the other modules to decide if we want to import or not?

yeah, just another way -- no strong opinion

@kaxil kaxil merged commit e345796 into main Nov 16, 2022
@kaxil kaxil deleted the fix-open-lineage-import branch November 16, 2022 12:22
kaxil pushed a commit that referenced this pull request Nov 16, 2022
@sunank200 @kaxil
Make openlineage an optional dependency (#1252)
02e5bd0 
closes: #1243

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
(cherry picked from commit e345796)
@phanikumv phanikumv mentioned this pull request Nov 17, 2022
Closed
@phanikumv phanikumv mentioned this pull request Jan 24, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kaxil kaxil kaxil approved these changes

@feluelle feluelle feluelle left review comments

@dimberman dimberman Awaiting requested review from dimberman dimberman is a code owner

@tatiana tatiana Awaiting requested review from tatiana tatiana is a code owner

@utkarsharma2 utkarsharma2 Awaiting requested review from utkarsharma2 utkarsharma2 is a code owner

@pankajastro pankajastro Awaiting requested review from pankajastro pankajastro is a code owner

@pankajkoti pankajkoti Awaiting requested review from pankajkoti

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Issue with openlineage import
3 participants
@sunank200 @kaxil @feluelle