feat(ingestion/bigquery): BigQuery Owner Label to Datahub Ownership #10047
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
ingestion
Comment on lines
299
to
309
| owner_lable_character_mapping: Dict[str, str] = Field( | ||
| default={}, | ||
| description="A mapping of bigquery owner label character to datahub owner character." | ||
| "Provided mapping will get added to default mapping.", | ||
| ) | ||
|
|
||
| owner_key_pattern: str = Field( | ||
| default="_owner_email", | ||
| description="A pattern which defines what identifies an owner label.", | ||
| ) | ||
|
|
There was a problem hiding this comment.
I'm looking at the BigQuery datahub source docs and seeing capture_table_labels_as_tags and think we should model this to be similar. So something like:
capture_table_owner_label_as_owner:
enabled: true
# we want them to be able to define a mapping, so our BQ label -> datahub owner ingestion is generic
label_character_mapping:
- "_": "."
- "-": "@"
- "__": "_"
- "--": "-"
- "_-": "#"
- "-_": " "
# we want them to be able to define what identifies an owner label, so it is also generic
owner_key_pattern: "_owner_email"
There was a problem hiding this comment.
…ownership_from_tags
hsheth2
reviewed
Mar 23, 2024
| @@ -111,5 +156,7 @@ def transform_aspect( | |||
| ), | |||
| ) | |||
| ) | |||
|
|
|||
| return None | |||
| if not self.config.replace_existing: | |||
There was a problem hiding this comment.
this doesn't make sense - we should always return aspect
There was a problem hiding this comment.
Even I thought so. But previous behavior was removing tag metadata from ingestion after extracting ownership from it.
hsheth2
reviewed
Mar 26, 2024
metadata-ingestion/src/datahub/ingestion/transformer/extract_ownership_from_tags.py
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/transformer/extract_ownership_from_tags.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/transformer/extract_ownership_from_tags.py
Outdated
Show resolved
Hide resolved
…into BigQuery-Owner-Label-to-Datahub-Ownership
hsheth2
reviewed
Mar 28, 2024
metadata-ingestion/src/datahub/ingestion/transformer/extract_ownership_from_tags.py
Outdated
Show resolved
Hide resolved
| ``` | ||
|
|
||
| So if we have input dataset tag like | ||
| - `urn:li:tag:dataset_owner_email:abc@email.com` | ||
| - `urn:li:tag:dataset_owner_email:xyz@email.com` |
There was a problem hiding this comment.
these have dataset_owner_email but the example is tag_pattern: "owner_email:" - why the mismatch?
There was a problem hiding this comment.
Because now the config is tag_pattern and not tag_prefix. We can provide pattern in this way as well.
hsheth2
approved these changes
Mar 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist