Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingest/lookml): correctly handle include directives from imported projects #7798

Merged

Conversation

hsheth2
Copy link
Collaborator

@hsheth2 hsheth2 commented Apr 11, 2023

The issue
When processing this situation: Base Project file -> includes External Project file A -> includes External Project file B, the second include directive was not being processed correctly. This was because we were looking for file B's include path in the base project rather than realizing that file A was in the external project and hence internal links should stay within the External Project.

How does this PR solve it?
We properly retain the context of a file so that we can process its includes correctly. We're doing that by deriving base_folder (now called resolved_project_folder) from the active project_name + any include directives.

What else is happening?

  • Because we now derive the base_folder from the base_projects_folder and the active project_name, we can remove the base_folder arg from the include resolver.
  • I couldn't find any docs for this, but it seems the include path is sometimes '/{project_name}/{path_within_project}' (normally cross-project refs start with //). I'm adding some edge case handling to process that correctly. To do that, we need the name of the base/root project root_project_name (if one exists), so we also do the plumbing to pipe that info into the include resolver.

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Apr 11, 2023
@jjoyce0510
Copy link
Collaborator

Can you add a PR description giving a high level overview of the changes that were required here? I'm having a hard time zooming out

@hsheth2
Copy link
Collaborator Author

hsheth2 commented Apr 12, 2023

@jjoyce0510 updated the PR desc with some docs about this change

Copy link
Contributor

@shirshanka shirshanka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jjoyce0510 jjoyce0510 merged commit 4f59169 into datahub-project:master Apr 13, 2023
@hsheth2 hsheth2 deleted the lookml-external-include-resolution branch April 13, 2023 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ingestion PR or Issue related to the ingestion of metadata
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants