Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(ingest): MongoDB schema inference #2546

Merged
merged 30 commits into from
May 14, 2021
Merged

feat(ingest): MongoDB schema inference #2546

merged 30 commits into from
May 14, 2021

Conversation

kevinhu
Copy link
Contributor

@kevinhu kevinhu commented May 12, 2021 •
edited
Loading

Adds schema inference capabilities for MongoDB:

  • Detects nested fields (concatenates with '.')
  • Detects if field is nullable (single MongoDB is schemaless, this translates to if the field is ever missing)
  • Includes sample_size option to scan a few documents rather than an entire collection

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable)

kevinhu and others added 22 commits May 10, 2021 16:30
@kevinhu
Rename tests
e9209d8
@kevinhu
Update mongo tests
94166df
@kevinhu
Add another mongo test case
a664d4c
@kevinhu
Sort MCE datasets and collections
fdeb4e5
@kevinhu
Add pymongo-schema scripts
c8295dd
@kevinhu
List collection schemas
77cc7dd
@kevinhu
Remove print
a5f66d4
@kevinhu
Merge branch 'master' of github.com:kevinhu/datahub into mongodb-sche…
99e3fcc 
…ma-ingestion
@kevinhu
Print schemas
cfbe3f1
@kevinhu
Merge branch 'linkedin:master' into mongodb-schema-ingestion
6358e02
@kevinhu
Add mixed type
74d81a1
@kevinhu
Clean up functions
ab10745
@kevinhu
Clean up Mongo inference functions
3611faf
@kevinhu
Add mongo types
1833311
@kevinhu
Basic type+nullability inference
4cf530e
@kevinhu
Mongo ingest
4a430e9
@kevinhu
Add sampling functionality to mongo ingestion
ade45c8
@kevinhu
Comments for mongo
ffc75e0
@kevinhu
Fix pymongo type string logging
af6fe46
@kevinhu
Merge branch 'master' of github.com:kevinhu/datahub into mongodb-sche…
2586f60 
…ma-ingestion
@kevinhu
Remove ete3 dependency
aaed234
@kevinhu
Remove logging import
8569913
@kevinhu kevinhu changed the title feat(ingest) MongodDB schema inference feat(ingest): MongodDB schema inference May 12, 2021
@kevinhu kevinhu changed the title feat(ingest): MongodDB schema inference feat(ingest): MongoDB schema inference May 13, 2021
@kevinhu kevinhu marked this pull request as ready for review May 13, 2021 17:29
hsheth2
hsheth2 requested changes May 13, 2021
View reviewed changes
Copy link
Collaborator

@hsheth2 hsheth2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking pretty good so far - just some small things

metadata-ingestion/README.md Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/mongodb.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/mongodb.py Show resolved Hide resolved
metadata-ingestion/tests/integration/ldap/ldap_mce_golden.json Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/mongodb.py Outdated Show resolved Hide resolved
hsheth2
hsheth2 approved these changes May 13, 2021
View reviewed changes
Copy link
Collaborator

@hsheth2 hsheth2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with one minor nit

metadata-ingestion/README.md Outdated Show resolved Hide resolved
shirshanka
shirshanka approved these changes May 14, 2021
View reviewed changes
Copy link
Contributor

@shirshanka shirshanka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@shirshanka shirshanka merged commit 5ab1cbb into datahub-project:master May 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shirshanka shirshanka shirshanka approved these changes

@hsheth2 hsheth2 hsheth2 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kevinhu @shirshanka @hsheth2