Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingest): unity-catalog - Unity catalog source #6292

Merged
merged 19 commits into from
Oct 30, 2022

Conversation

treff7es
Copy link
Contributor

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

…ata/datahub-fork into master+acr-4552-mvp-unity-catalog
- Stateful ingestion
- Filtering
- More verbose reporting
- Domain support
- Connection test support
- Complex type support
Moving config into separate file
@treff7es treff7es changed the title Unity catalog support feat(ingest): unity-catalog -Unity catalog source Oct 26, 2022
@treff7es treff7es changed the title feat(ingest): unity-catalog -Unity catalog source feat(ingest): unity-catalog - Unity catalog source Oct 26, 2022
@github-actions github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Oct 26, 2022
@github-actions
Copy link

github-actions bot commented Oct 26, 2022

Unit Test Results (build & test)

599 tests   595 ✔️  11m 16s ⏱️
149 suites      4 💤
149 files        0

Results for commit 96bd74c.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Oct 26, 2022

Unit Test Results (metadata ingestion)

       8 files  ±  0         8 suites  ±0   54m 34s ⏱️ - 2m 17s
   756 tests +  8     747 ✔️ +  2  3 💤 ±0  1 +1  5 🔥 +5 
1 514 runs  +16  1 502 ✔️ +10  6 💤 ±0  1 +1  5 🔥 +5 

For more details on these failures and errors, see this check.

Results for commit 96bd74c. ± Comparison against base commit 46453e4.

♻️ This comment has been updated with latest results.

@treff7es
Copy link
Contributor Author

treff7es commented Oct 28, 2022

CleanShot 2022-10-28 at 19 02 57@2x

CleanShot 2022-10-28 at 19 03 18@2x

CleanShot 2022-10-28 at 19 03 26@2x

CleanShot 2022-10-28 at 19 03 50@2x

catalog: str


class UnitySchemaKey(CatalogKey):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a big fan of integration specific classes being added to mcp_builder.py ... we have accumulated a few and need to remove these later IMO

Copy link
Contributor

@shirshanka shirshanka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@shirshanka shirshanka merged commit 0dc2d6a into master Oct 30, 2022
@shirshanka shirshanka deleted the master+acr-4552-mvp-unity-catalog branch October 30, 2022 04:44
anthonyhungnguyen pushed a commit to anthonyhungnguyen/zlp-datahub that referenced this pull request Nov 8, 2022
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
cccs-tom pushed a commit to CybercentreCanada/datahub that referenced this pull request Nov 18, 2022
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
cccs-tom pushed a commit to CybercentreCanada/datahub that referenced this pull request Nov 18, 2022
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ingestion PR or Issue related to the ingestion of metadata
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants