fix(ingest): bigquery-beta - turning sql parsing off in lineage extraction #6163
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… significantly the lineage generation and it was not precise enough Collecting report metrics for lineage per project and not globally Fixing table pattern in lineage extraction
github-actions
bot
added
the
ingestion
Unit Test Results (metadata ingestion) 8 files + 1 8 suites +1 58m 33s ⏱️ + 9m 54s Results for commit d98216c. ± Comparison against base commit e9f6154. This pull request removes 1 test.♻️ This comment has been updated with latest results. |
hsheth2
reviewed
Oct 11, 2022
| # The inheritance hierarchy is wonky here, but these options need modifications. | ||
| project_id: Optional[str] = Field( | ||
| default=None, | ||
| description="[deprecated] Use project_id_pattern instead.", | ||
| ) | ||
| storage_project_id: None = Field(default=None, exclude=True) | ||
|
|
||
| lineage_use_sql_parser: bool = Field( | ||
| default=False, | ||
| description="Experimental. Use sql parser to resolve view/table lineage. If there is a view being referenced then bigquery sends both the view as well as underlying tablein the references. There is no distinction between direct/base objects accessed. So doing sql parsing to ensure we only use direct objects accessed for lineage.", |
There was a problem hiding this comment.
Suggested change
| description="Experimental. Use sql parser to resolve view/table lineage. If there is a view being referenced then bigquery sends both the view as well as underlying tablein the references. There is no distinction between direct/base objects accessed. So doing sql parsing to ensure we only use direct objects accessed for lineage.", | |
| description="Experimental. Use sql parser to resolve view/table lineage. If there is a view being referenced then BigQuery records both the view and the underlying tables as lineage. Enabling SQL parsing allows us to only include the views/tables that were actually accessed by a query.", |
Comment on lines
+363
to
+365
| self.report.num_total_audit_entries[event.project_id] = ( | ||
| self.report.num_total_audit_entries.get(event.project_id, 0) + 1 | ||
| ) |
There was a problem hiding this comment.
Ideally the TopKDict would be a subclass of defaultdict. That'd let this code be simplified
Suggested change
| self.report.num_total_audit_entries[event.project_id] = ( | |
| self.report.num_total_audit_entries.get(event.project_id, 0) + 1 | |
| ) | |
| self.report.num_total_audit_entries[event.project_id] += 1 |
| dict_as_tuples = sorted_tuples[:10] | ||
| trimmed_dict = {k: v for k, v in dict_as_tuples} | ||
| trimmed_dict[f"... top(10) of total {len(big_dict)} entries"] = "" | ||
| print(f"Dropping entries {sorted_tuples[11:]}") |
|
|
||
| def __init__(self, top_k: int = 10) -> None: | ||
| super().__init__() | ||
| self.top_k = 10 |
There was a problem hiding this comment.
Suggested change
| self.top_k = 10 | |
| self.top_k = top_k |
| @@ -0,0 +1,36 @@ | |||
| from typing import Any, Dict, TypeVar, Union | |||
|
|
|||
| T = TypeVar("T") | |||
There was a problem hiding this comment.
it doesn't look like this is used anywhere?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist