Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(ingest/bigquery): use correct row count in null count profiling c… #9123

Merged
merged 4 commits into from
Nov 2, 2023
Merged

fix(ingest/bigquery): use correct row count in null count profiling c… #9123

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0ce4d1e
fix(ingest/bigquery): use correct row count in null count profiling c…
mayurinehate Oct 27, 2023
f3cd6d4
update only local row count assignment
mayurinehate Oct 30, 2023
df25dcc
Merge branch 'master' into master+ing-410-bigquery-sampled-profiling-fix
hsheth2 Oct 31, 2023
0e6d5a3
Merge branch 'master' into master+ing-410-bigquery-sampled-profiling-fix
anshbansal Nov 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
9 changes: 8 additions & 1 deletion metadata-ingestion/src/datahub/ingestion/source/ge_data_profiler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -781,7 +781,7 @@ def update_dataset_batch_use_sampling(self, profile: DatasetProfileClass) -> Non
sample_pc = 100 * self.config.sample_size / profile.rowCount
sql = (
f"SELECT * FROM {str(self.dataset._table)} "
+ f"TABLESAMPLE SYSTEM ({sample_pc:.3f} percent)"
+ f"TABLESAMPLE SYSTEM ({sample_pc:.8f} percent)"
)
temp_table_name = create_bigquery_temp_table(
self,
Expand All @@ -793,6 +793,13 @@ def update_dataset_batch_use_sampling(self, profile: DatasetProfileClass) -> Non
self.dataset._table = sa.text(temp_table_name)
logger.debug(f"Setting table name to be {self.dataset._table}")

# We can alternatively use `self._get_dataset_rows(profile)` to get
# exact count of rows in sample, as actual rows involved in sample
# may be slightly different (more or less) than configured `sample_size`.
# However not doing so to start with, as that adds another query overhead
# plus approximate metrics should work for sampling based profiling.
profile.rowCount = self.config.sample_size
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imo the row count should still ultimately reflect the full number of rows, and the size of the sample should be stored in the partition spec

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The actual table's rowCount here is always overridden in bigquery source from information_schema, whenever partitionSpec is set. partitionSpec is set whenever limit or sampling is used in profiling or only latest partition is used for partitioned tables. The computed rowCount in ge_data_profiler.py is used primarily for column level profiling computation only. (this is also the case for snowflake and redshift)
Here - https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/sql/sql_generic_profiler.py#L112

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to handling this only for null counts computation and not updating local profile aspect. Attempted changing profile aspect to make sure that we don't run into this for any other calculation - maybe in future.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup that might result in more understandable code

At minimum, we need a comment here explaining why it's ok to override, but ideally we just tweak the null count / unique percentage calculations


if (
profile.partitionSpec
and profile.partitionSpec.type == PartitionTypeClass.FULL_TABLE
Expand Down