[FEATURE] Implement the "column.standard_deviation" metric for sqlite database #5338

alexsherstinsky · 2022-06-17T21:00:03Z

Please annotate your PR title to describe what the PR does, then give a brief bulleted description of your PR below. PR titles should begin with [BUGFIX], [FEATURE], [DOCS], or [MAINTENANCE]. If a new feature introduces breaking changes for the Great Expectations API or configuration files, please also add [BREAKING]. You can read about the tags in our contributor checklist.

Changes proposed in this pull request:

JIRA: GREAT-586

After submitting your PR, CI checks will run and @cla-bot will check for your CLA signature.

For a PR with nontrivial changes, we review with both design-centric and code-centric lenses.

In a design review, we aim to ensure that the PR is consistent with our relationship to the open source community, with our software architecture and abstractions, and with our users' needs and expectations. That review often starts well before a PR, for example in github issues or slack, so please link to relevant conversations in notes below to help reviewers understand and approve your PR more quickly (e.g. closes #123).

Previous Design Review notes:

Definition of Done

Please delete options that are not relevant.

My code follows the Great Expectations style guide
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added unit tests where applicable and made sure that new and existing tests are passing.
I have run any local integration tests and made sure that nothing is broken.

Thank you for submitting!

…4/GREAT-999/alexsherstinsky/rule_based_profiler/data_assistant/data_assistant_result/move_getting_expectation_suite_to_expectation_suite_class-2022_06_15-170

…/alexsherstinsky/rule_based_profiler/remove_json_serialize_directive_and_add_raw_parameter_builder_computation_results_to_json_serialized_results-2022_06_15-170

…ror-changelog' into maintenance/GREAT-467/GREAT-464/GREAT-999/alexsherstinsky/rule_based_profiler/remove_json_serialize_directive_and_add_raw_parameter_builder_computation_results_to_json_serialized_results-2022_06_15-170

…ky/rule_based_profiler/remove_json_serialize_directive_and_add_raw_parameter_builder_computation_results_to_json_serialized_results-2022_06_15-170' into pre_pr-prototype/maintenance/GREAT-467/GREAT-464/GREAT-1000/alexsherstinsky/rule_based_profiler/data_assistant/onboarding_data_assistant/performance_improvements-2022_06_15-171

…exsherstinsky/rule_based_profiler/enable_numeric_metric_range_multibatch_parameter_builder_to_use_evaluation_dependencies-2022_06_16-171

…EAT-464/GREAT-1000/alexsherstinsky/rule_based_profiler/data_assistant/onboarding_data_assistant/performance_improvements-2022_06_15-171

…rule_based_profiler/enable_numeric_metric_range_multibatch_parameter_builder_to_use_evaluation_dependencies-2022_06_16-171' into pre_pr-prototype/maintenance/GREAT-467/GREAT-464/GREAT-1000/alexsherstinsky/rule_based_profiler/data_assistant/onboarding_data_assistant/performance_improvements-2022_06_15-171

…/maintenance/GREAT-467/GREAT-464/GREAT-1000/alexsherstinsky/rule_based_profiler/data_assistant/onboarding_data_assistant/performance_improvements-2022_06_15-171

…lder throughout the codebase.

…EAT-464/GREAT-1000/alexsherstinsky/rule_based_profiler/data_assistant/onboarding_data_assistant/performance_improvements-2022_06_15-171

ghost · 2022-06-17T21:01:13Z

👇 Click on the image for a new way to code review

Make big changes easier — review code in small groups of related files
Know where to start — see the whole change at a glance
Take a code tour — explore the change with an interactive tour
Make comments and review — all fully sync’ed with github

Try it now!

Legend

netlify · 2022-06-17T21:02:00Z

✅ Deploy Preview for niobium-lead-7998 ready!

Name	Link
🔨 Latest commit	`246be4d`
🔍 Latest deploy log	https://app.netlify.com/sites/niobium-lead-7998/deploys/62ae2905651ae700087e0164
😎 Deploy Preview	https://deploy-preview-5338--niobium-lead-7998.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

Shinnnyshinshin · 2022-06-17T21:09:00Z

great_expectations/expectations/metrics/column_aggregate_metrics/column_mean.py

@@ -25,7 +25,7 @@ def _pandas(cls, column, **kwargs):
    def _sqlalchemy(cls, column, **kwargs):
        """SqlAlchemy Mean Implementation"""
        # column * 1.0 needed for correct calculation of avg in MSSQL
-        return sa.func.avg(column * 1.0)
+        return sa.func.avg(1.0 * column)


Just curious: does this make a difference in the results of the calculation?

@Shinnnyshinshin No -- only sense of aesthetics on my part. Thanks!

…_standard_deviation_metric_for_sqlite_database-2022_06_17-176

Shinnnyshinshin

Really appreciate you digging into this @alexsherstinsky Thank you very very much. LGTM

Shinnnyshinshin · 2022-06-17T21:25:10Z

great_expectations/expectations/metrics/column_aggregate_metrics/column_standard_deviation.py

+        elif _dialect.name.lower() == "sqlite":
+            mean = _metrics["column.mean"]
+            nonnull_row_count = _metrics["column_values.null.unexpected_count"]
+            standard_deviation = sa.func.sqrt(
+                sa.func.sum((1.0 * column - mean) * (1.0 * column - mean))
+                / ((1.0 * nonnull_row_count) - 1.0)
+            )
        else:


Shinnnyshinshin · 2022-06-17T21:27:00Z

great_expectations/self_check/util.py

@@ -1496,7 +1496,6 @@ def candidate_test_is_on_temporary_notimplemented_list_cfe(context, expectation_
            "expect_column_values_to_be_dateutil_parseable",
            "expect_column_values_to_be_json_parseable",
            "expect_column_values_to_match_json_schema",
-            "expect_column_stdev_to_be_between",


Whoohoo. Just explicitly stating that this enables the expect_column_stdev_to_be_between Expectation tests for sql dialects (include sqlite). So additional unittests are not needed for this PR

kenwade4

Looks great and everything is passing for expect_column_stdev_to_be_between in the build_gallery.py script! Thanks!

…ture/GREAT-953/migration-part2-refactor-usage-stats-opt * feature/GREAT-953/migration-part1-move-to-abc: Update base_data_context.py [MAINTENANCE] Update release schedule JSON (#5349) [DOCS] DOC-337 automate updates to the version information displayed in the getting started tutorial. (#5348) Maintenance/great 761/great 1010/great 1011/alexsherstinsky/rule based profiler/data assistant/include only essential public methods in data assistant dispatcher class 2022 06 21 177 (#5351) [FEATURE] Implement the "column.standard_deviation" metric for sqlite database (#5338) [BUGFIX] Fix for failing Expectation test in `cloud_db_integration` pipeline (#5321)

Alex Sherstinsky and others added 30 commits June 15, 2022 15:41

remove json_serialize

04f295f

Merge remote-tracking branch 'upstream/develop' into feature/GREAT-46…

67ec0a4

…4/GREAT-999/alexsherstinsky/rule_based_profiler/data_assistant/data_assistant_result/move_getting_expectation_suite_to_expectation_suite_class-2022_06_15-170

Merge branch 'develop' into maintenance/GREAT-467/GREAT-464/GREAT-999…

8f68e80

…/alexsherstinsky/rule_based_profiler/remove_json_serialize_directive_and_add_raw_parameter_builder_computation_results_to_json_serialized_results-2022_06_15-170

Merge branch 'develop' into maintenance/GREAT-467/GREAT-464/GREAT-999…

c0d9e63

…/alexsherstinsky/rule_based_profiler/remove_json_serialize_directive_and_add_raw_parameter_builder_computation_results_to_json_serialized_results-2022_06_15-170

fixes netlify issue by removing tag

388cd0a

variables cleanup

9cdb93f

Merge branch 'develop' into maintenance/GREAT-467/GREAT-464/GREAT-999…

3d03098

…/alexsherstinsky/rule_based_profiler/remove_json_serialize_directive_and_add_raw_parameter_builder_computation_results_to_json_serialized_results-2022_06_15-170

clean up

318b73e

adding name back

a23de33

enable numeric range parameter builder to use evaluation dependencies

db7ba1f

merge

1440137

clean up

60d1c8b

clean up

d6ec2a3

clean up

c46cd82

clean up

251581e

clean up type hints

62e20ea

Merge branch 'develop' into feature/GREAT-467/GREAT-464/GREAT-1001/al…

3f03e7b

…exsherstinsky/rule_based_profiler/enable_numeric_metric_range_multibatch_parameter_builder_to_use_evaluation_dependencies-2022_06_16-171

Merge branch 'develop' into pre_pr-prototype/maintenance/GREAT-467/GR…

0fce14a

…EAT-464/GREAT-1000/alexsherstinsky/rule_based_profiler/data_assistant/onboarding_data_assistant/performance_improvements-2022_06_15-171

Merge remote-tracking branch 'upstream/develop' into pre_pr-prototype…

ce9cd7a

…/maintenance/GREAT-467/GREAT-464/GREAT-1000/alexsherstinsky/rule_based_profiler/data_assistant/onboarding_data_assistant/performance_improvements-2022_06_15-171

instrumenting

0738f1d

clean up

fd61254

Standardize instantiation of NumericMetricRangeMultibatchParameterBui…

283e1f1

…lder throughout the codebase.

Standardize instantiation of NumericMetricRangeMultibatchParameterBui…

9bcbfaf

…lder throughout the codebase.

Merge branch 'develop' into pre_pr-prototype/maintenance/GREAT-467/GR…

164f80a

…EAT-464/GREAT-1000/alexsherstinsky/rule_based_profiler/data_assistant/onboarding_data_assistant/performance_improvements-2022_06_15-171

use prior ParameterBuilder computations efficiently

4f9b404

merge

e6a96a4

Merge branch 'develop' into pre_pr-prototype/maintenance/GREAT-467/GR…

89c1313

…EAT-464/GREAT-1000/alexsherstinsky/rule_based_profiler/data_assistant/onboarding_data_assistant/performance_improvements-2022_06_15-171

alexsherstinsky requested a review from kenwade4 June 17, 2022 21:00

Shinnnyshinshin reviewed Jun 17, 2022

View reviewed changes

Alex Sherstinsky added 2 commits June 17, 2022 14:11

comment

af4d984

comment

f9fd591

alexsherstinsky disabled auto-merge June 17, 2022 21:13

alexsherstinsky enabled auto-merge (squash) June 17, 2022 21:14

Merge develop into feature/GREAT-586/alexsherstinsky/implement_column…

5a349c0

…_standard_deviation_metric_for_sqlite_database-2022_06_17-176

alexsherstinsky disabled auto-merge June 17, 2022 21:18

alexsherstinsky enabled auto-merge (squash) June 17, 2022 21:18

Shinnnyshinshin approved these changes Jun 17, 2022

View reviewed changes

alexsherstinsky mentioned this pull request Jun 17, 2022

[BUGFIX] For expect_column_stdev_to_be_between, don't run tests on sqlite #4197

Closed

2 tasks

kenwade4 approved these changes Jun 17, 2022

View reviewed changes

alexsherstinsky disabled auto-merge June 18, 2022 01:18

Alex Sherstinsky added 3 commits June 17, 2022 21:57

sqlite3

91b665b

sqlite3

3363b9b

sqlite3

2939a59

alexsherstinsky enabled auto-merge (squash) June 18, 2022 05:26

alexsherstinsky disabled auto-merge June 18, 2022 05:26

alexsherstinsky enabled auto-merge (squash) June 18, 2022 05:26

revert

df43186

alexsherstinsky disabled auto-merge June 18, 2022 06:52

Alex Sherstinsky added 3 commits June 18, 2022 01:39

custom sqrt function for sqlite

09278c9

attempting to define sqrt in sqlite

ec5f2e2

attempting to define sqrt in sqlite

246be4d

alexsherstinsky enabled auto-merge (squash) June 18, 2022 21:27

alexsherstinsky merged commit f9a2127 into develop Jun 18, 2022

alexsherstinsky deleted the feature/GREAT-586/alexsherstinsky/implement_column_standard_deviation_metric_for_sqlite_database-2022_06_17-176 branch June 18, 2022 22:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] Implement the "column.standard_deviation" metric for sqlite database #5338

[FEATURE] Implement the "column.standard_deviation" metric for sqlite database #5338

alexsherstinsky commented Jun 17, 2022

ghost commented Jun 17, 2022 •

edited by ghost

Loading

netlify bot commented Jun 17, 2022 •

edited

Loading

Shinnnyshinshin Jun 17, 2022

alexsherstinsky Jun 17, 2022

Shinnnyshinshin left a comment

Shinnnyshinshin Jun 17, 2022

Shinnnyshinshin Jun 17, 2022

kenwade4 left a comment

[FEATURE] Implement the "column.standard_deviation" metric for sqlite database #5338

[FEATURE] Implement the "column.standard_deviation" metric for sqlite database #5338

Conversation

alexsherstinsky commented Jun 17, 2022

Previous Design Review notes:

Definition of Done

ghost commented Jun 17, 2022 • edited by ghost Loading

Legend

netlify bot commented Jun 17, 2022 • edited Loading

✅ Deploy Preview for niobium-lead-7998 ready!

Shinnnyshinshin Jun 17, 2022

Choose a reason for hiding this comment

alexsherstinsky Jun 17, 2022

Choose a reason for hiding this comment

Shinnnyshinshin left a comment

Choose a reason for hiding this comment

Shinnnyshinshin Jun 17, 2022

Choose a reason for hiding this comment

Shinnnyshinshin Jun 17, 2022

Choose a reason for hiding this comment

kenwade4 left a comment

Choose a reason for hiding this comment

ghost commented Jun 17, 2022 •

edited by ghost

Loading

netlify bot commented Jun 17, 2022 •

edited

Loading