Skip to content

SCRUM-1053-explore profile reports the value domain of a low-cardinality non-PII… - #238

Open
emnarahmouni-exmergo wants to merge 5 commits into
mainfrom
SCRUM-1053-ISSUE-203
Open

SCRUM-1053-explore profile reports the value domain of a low-cardinality non-PII…#238
emnarahmouni-exmergo wants to merge 5 commits into
mainfrom
SCRUM-1053-ISSUE-203

Conversation

@emnarahmouni-exmergo

Copy link
Copy Markdown
Contributor

Closes : #203
ColumnProfile gains a value_domain field (ValueDomain: capped values by frequency + elided count), populated by a new _probe_value_domains step in explore/profile.py.
Eligibility (all required): no PII flag at any confidence, distinct count ≤ 25 and ≤ 10% of non-null rows, not a candidate key (single-column or proven composite member).
New adapter capability value_domain_counts, implemented for DuckDB and BigQuery this pass (one batched statement: capped top-K by frequency + exact distinct-group count per column). Other connectors don't implement it yet and degrade to reporting no domain, same as any adapter that predates this feature.
Eligibility is decided from the approximate pre-scan distinct count (cost control); the reported domain is built from the exact count the probe returns if approximation under-estimated but the true count still clears the fraction bar, the result is capped with an honest elided count instead of being dropped; only a true count that breaks the fraction bar is dropped entirely.
BigQuery: cost-gated like every other optional escalation (dry-run, floor at the per-query minimum, try_charge, degrade to {} + a table note on insufficient budget); profile_estimate reserves for it up front.
before the fix :
image
After fixing :
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

explore profile: report the value domain of a low-cardinality non-PII column

1 participant