Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(ingest/looker): Record observed lineage timestamps for Looker and LookML sources #7735

Merged
merged 4 commits into from
Aug 24, 2023
Merged

feat(ingest/looker): Record observed lineage timestamps for Looker and LookML sources #7735

merged 4 commits into from
Aug 24, 2023

Conversation

ANich
Copy link
Contributor

@ANich ANich commented Mar 31, 2023 •
edited

Adds timestamps to the Looker and LookML sources for lineage between Explores and Views as well as between Views and SQL tables/views.

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
    - [ ] Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
    - [ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
    - [ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Mar 31, 2023
@ANich ANich force-pushed the looker_lineage_timestamps branch 2 times, most recently from e66adde to 1224d2e Compare March 31, 2023 16:18
@ANich ANich marked this pull request as ready for review March 31, 2023 16:18
hsheth2
hsheth2 requested changes Apr 18, 2023
View reviewed changes
metadata-ingestion/src/datahub/ingestion/source/looker/looker_common.py Outdated
@@ -841,6 +843,10 @@ def _to_metadata_events( # noqa: C901
UpstreamClass(
dataset=view_urn,
type=DatasetLineageTypeClass.VIEW,
auditStamp=AuditStamp(
time=int(observed_lineage_ts.timestamp() * 1000),
actor="urn:li:corpuser:datahub",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it'd be good to extract this out into a constant

metadata-ingestion/src/datahub/ingestion/source/looker/looker_common.py Outdated
@@ -21,6 +21,7 @@
)

import pydantic
from datahub.metadata.com.linkedin.pegasus2avro.common import AuditStamp
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like linter doesn't like the import sorting here

@laulpogan
Copy link
Contributor

@ANich - we haven't seen movement on this PR for a little while- are you still interested in contributing? If so, please let us know within the week, or we'll need to close this one for inactivity.

@anshbansal anshbansal added the community-contribution PR or Issue raised by member(s) of DataHub Community label Jun 23, 2023
@ANich
Copy link
Contributor Author

ANich commented Jul 19, 2023

@ANich - we haven't seen movement on this PR for a little while- are you still interested in contributing? If so, please let us know within the week, or we'll need to close this one for inactivity.

Hey @laulpogan @anshbansal I'll pick this back up in the next couple days if that's okay. Will ping for a re-review

@ANich ANich force-pushed the looker_lineage_timestamps branch from 810923f to 2eb7aa2 Compare July 21, 2023 14:29
@ANich ANich requested a review from hsheth2 July 21, 2023 14:31
hsheth2
hsheth2 reviewed Aug 15, 2023
View reviewed changes
Copy link
Collaborator

@hsheth2 hsheth2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should to add tz to now() calls, but otherwise lgtm

metadata-ingestion/src/datahub/ingestion/source/looker/looker_common.py Outdated
@@ -786,6 +789,7 @@ def _to_metadata_events( # noqa: C901
if self.upstream_views is not None:
assert self.project_name is not None
upstreams = []
observed_lineage_ts = datetime.datetime.now()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
observed_lineage_ts = datetime.datetime.now()
observed_lineage_ts = datetime.datetime.now(tz=timezone.utc)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated! Thanks for the review

metadata-ingestion/src/datahub/ingestion/source/looker/lookml_source.py Outdated
@@ -1615,11 +1617,16 @@ def _get_upstream_lineage(

# Generate the upstream + fine grained lineage objects.
upstreams = []
observed_lineage_ts = datetime.now()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs tz

@hsheth2 hsheth2 added the merge-pending-ci A PR that has passed review and should be merged once CI is green. label Aug 17, 2023
@anshbansal anshbansal merged commit bcef25a into datahub-project:master Aug 24, 2023
49 of 51 checks passed
@anshbansal
Copy link
Collaborator

cypress test failures unrelated so merging.

@ANich ANich deleted the looker_lineage_timestamps branch August 24, 2023 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hsheth2 hsheth2 hsheth2 approved these changes

@mayurinehate mayurinehate Awaiting requested review from mayurinehate

Assignees
No one assigned
Labels
community-contribution PR or Issue raised by member(s) of DataHub Community ingestion PR or Issue related to the ingestion of metadata merge-pending-ci A PR that has passed review and should be merged once CI is green.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@ANich @laulpogan @anshbansal @hsheth2