Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingest/jsonschema) enable schema-aware JsonSchemaTranslator #8971

Merged
merged 3 commits into from Oct 30, 2023

Conversation

KulykDmytro
Copy link
Contributor

@KulykDmytro KulykDmytro commented Oct 7, 2023

Changes

Picking up schema-validator class based on one defined in schema instead of using Draft7 everywhere

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

closes #8970

@github-actions github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Oct 7, 2023
@KulykDmytro KulykDmytro changed the title feat(injest/jsonschema) enable schema-aware JSON parsing feat(injest/jsonschema) enable schema-aware JsonSchemaTranslator Oct 7, 2023
@KulykDmytro KulykDmytro changed the title feat(injest/jsonschema) enable schema-aware JsonSchemaTranslator feat(ingest/jsonschema) enable schema-aware JsonSchemaTranslator Oct 7, 2023
@hsheth2
Copy link
Collaborator

hsheth2 commented Oct 10, 2023

@KulykDmytro not super familiar with the jsonschema library - what exactly does this do?

@KulykDmytro
Copy link
Contributor Author

KulykDmytro commented Oct 25, 2023

@KulykDmytro not super familiar with the jsonschema library - what exactly does this do?

@hsheth2 It handles json-schema operating/validation
BTW datahub is using it underneath but in narrowed way (hardcoding schema Draft to v7)

we faced same issue in our internal projects (validate kafka json data over json schemas) - so proposing solutions we have now.

PS: additionally added some changes description into first post

@maggiehays maggiehays added the hacktoberfest-accepted Acceptance for hacktoberfest https://hacktoberfest.com/participation/ label Oct 26, 2023
@hsheth2
Copy link
Collaborator

hsheth2 commented Oct 27, 2023

@KulykDmytro oh I see - this is enabling us to detect the jsonschema spec version, and use the appropriate validator

@hsheth2 hsheth2 added the merge-pending-ci A PR that has passed review and should be merged once CI is green. label Oct 27, 2023
@hsheth2 hsheth2 merged commit 758ed47 into datahub-project:master Oct 30, 2023
54 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hacktoberfest-accepted Acceptance for hacktoberfest https://hacktoberfest.com/participation/ ingestion PR or Issue related to the ingestion of metadata merge-pending-ci A PR that has passed review and should be merged once CI is green.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable draft-aware for JsonSchemaTranslator
3 participants