-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[MAINTENANCE] Add proper unit tests for Column Histogram metric and use Column Value Partitioner in OnboardingDataAssistant #5267
Conversation
…ky/rule_based_profiler/data_assistant/metrics/column_histogram_metric_must_support_integer_bins_for_all_execution_engine_options-2022_06_07-163
✅ Deploy Preview for niobium-lead-7998 ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
👇 Click on the image for a new way to code review
Legend |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A couple of questions!
@@ -280,8 +280,10 @@ def _build_numeric_columns_rule() -> Rule: | |||
|
|||
# Step-2: Declare "ParameterBuilder" for every metric of interest. | |||
|
|||
column_histogram_metric_multi_batch_parameter_builder_for_metrics: ParameterBuilder = DataAssistant.commonly_used_parameter_builders.get_column_histogram_metric_multi_batch_parameter_builder( | |||
json_serialize=True | |||
column_partition_parameter_builder_for_metrics: ParameterBuilder = DataAssistant.commonly_used_parameter_builders.build_partition_parameter_builder( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What exactly makes column.partition
an improvement over using column.histogram
for continuous data? On the face of it, it would seem like they do the same thing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@NathanFarmer The PartitionParameterBuilder
is better, because it has the logic to decide whether or not the underlying data is categorical or finality-quantized (as an approximation to "continuous"). This optimizes the histogram calculations for best interpretation. In the histogram case, we produce probability distribution like output (and care about smoothing the tails). In the categorical case, we produce the probability mass function like output. Thanks!
@@ -661,7 +664,7 @@ def _build_datetime_columns_rule() -> Rule: | |||
"mostly": 1.0, | |||
"strict_min": False, | |||
"strict_max": False, | |||
"bins": 10, | |||
"allow_relative_error": "linear", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This parameter seems to be important, but why can't this just be added to column.histogram
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@NathanFarmer Because it would contravene the original API, whereby the column.histogram
metric accepts the list, containing the bin boundaries (the bins
parameter). So I had to revert in order to stay true to the original intent. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me once that unused import gets removed!
…ky/rule_based_profiler/data_assistant/metrics/column_histogram_metric_must_support_integer_bins_for_all_execution_engine_options-2022_06_07-163
Scope
This contribution accomplishes several related tasks:
PartitionParameterBuilder
inOnboardingDataAssistant
due to its generality of handling histogram style requirements for both, categorical as well as finely-quantized (as an approximation to "continuous") column data values.DataAssistant
unit tests to handle the availableParameterNode
keys robustly.Please annotate your PR title to describe what the PR does, then give a brief bulleted description of your PR below. PR titles should begin with [BUGFIX], [FEATURE], [DOCS], or [MAINTENANCE]. If a new feature introduces breaking changes for the Great Expectations API or configuration files, please also add [BREAKING]. You can read about the tags in our contributor checklist.
Changes proposed in this pull request:
After submitting your PR, CI checks will run and @cla-bot will check for your CLA signature.
For a PR with nontrivial changes, we review with both design-centric and code-centric lenses.
In a design review, we aim to ensure that the PR is consistent with our relationship to the open source community, with our software architecture and abstractions, and with our users' needs and expectations. That review often starts well before a PR, for example in github issues or slack, so please link to relevant conversations in notes below to help reviewers understand and approve your PR more quickly (e.g.
closes #123
).Previous Design Review notes:
Definition of Done
Please delete options that are not relevant.
Thank you for submitting!