[FEATURE] Replace UserConfigurableProfiler with OnboardingDataAssistant in "CLI suite new --profile" Jupyter Notebooks #5236

alexsherstinsky · 2022-06-03T00:06:00Z

Scope

Replace UserConfigurableProfiler with OnboardingDataAssistant run() invocation in Jupyter notebooks associated with executing "CLI suite new --profile" on the command line.
Update unit tests.
Clean up BaseNotebookRenderer interfaces and extend these corrections to the subclasses thereof.
Fix the bug in processing the column-type directives, allowed as part of DataAssistant run() invocations.
Fix type hints usage in CLI-Suite implementation.
Replace UserConfigurableProfiler with OnboardingDataAssistant run() invocation in DocuSaurus "Getting Started" Jupyter notebook.
Minor clean up (typos, etc.).

Please annotate your PR title to describe what the PR does, then give a brief bulleted description of your PR below. PR titles should begin with [BUGFIX], [FEATURE], [DOCS], or [MAINTENANCE]. If a new feature introduces breaking changes for the Great Expectations API or configuration files, please also add [BREAKING]. You can read about the tags in our contributor checklist.

Changes proposed in this pull request:

JIRA: GREAT-761/GREAT-932

After submitting your PR, CI checks will run and @cla-bot will check for your CLA signature.

For a PR with nontrivial changes, we review with both design-centric and code-centric lenses.

In a design review, we aim to ensure that the PR is consistent with our relationship to the open source community, with our software architecture and abstractions, and with our users' needs and expectations. That review often starts well before a PR, for example in github issues or slack, so please link to relevant conversations in notes below to help reviewers understand and approve your PR more quickly (e.g. closes #123).

Previous Design Review notes:

Definition of Done

Please delete options that are not relevant.

My code follows the Great Expectations style guide
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added unit tests where applicable and made sure that new and existing tests are passing.
I have run any local integration tests and made sure that nothing is broken.

Thank you for submitting!

…flows.

…ky/rule_based_profiler/replace_user_configurable_profiler_with_onboarding_data_assistant_in_cli_suite_new-2022_06_02-158

netlify · 2022-06-03T00:06:05Z

✅ Deploy Preview for niobium-lead-7998 ready!

Name	Link
🔨 Latest commit	`3616e8f`
🔍 Latest deploy log	https://app.netlify.com/sites/niobium-lead-7998/deploys/6299af584a0dd10008b2ade1
😎 Deploy Preview	https://deploy-preview-5236--niobium-lead-7998.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

ghost · 2022-06-03T00:09:19Z

👇 Click on the image for a new way to code review

Make big changes easier — review code in small groups of related files
Know where to start — see the whole change at a glance
Take a code tour — explore the change with an interactive tour
Make comments and review — all fully sync’ed with github

Try it now!

Legend

Shinnnyshinshin

LGTM Thank you sir! Appreciate the synchronous review.

Shinnnyshinshin · 2022-06-03T00:44:54Z

tests/integration/docusaurus/tutorials/getting-started/getting_started.py

@@ -3,8 +3,8 @@
 import great_expectations as ge
 from great_expectations.checkpoint import SimpleCheckpoint
 from great_expectations.core.batch import BatchRequest
-from great_expectations.profile.user_configurable_profiler import (
-    UserConfigurableProfiler,


Shinnnyshinshin · 2022-06-03T00:45:23Z

great_expectations/render/renderer/v3/suite_edit_notebook_renderer.py

-        batch_request: Optional[
-            Union[str, Dict[str, Union[str, int, Dict[str, Any]]]]
-        ] = None,
+        batch_request: Optional[Union[str, Dict[str, Any]]] = None,


This brings me joy

Shinnnyshinshin · 2022-06-03T00:45:52Z

great_expectations/render/renderer/v3/suite_edit_notebook_renderer.py

-            Union[str, Dict[str, Union[str, int, Dict[str, Any]]]]
-        ] = None,
-    ) -> nbformat.NotebookNode:
+    def render(self, **kwargs: dict) -> nbformat.NotebookNode:


These are kwargs because subclasses require different parameters

Shinnnyshinshin · 2022-06-03T00:49:08Z

great_expectations/rule_based_profiler/data_assistant/data_assistant.py

        meta={
            "profiler_details": f"{column_values_attribute_mean_unexpected_value_multi_batch_parameter_builder_for_validations.fully_qualified_parameter_name}.{FULLY_QUALIFIED_PARAMETER_NAME_METADATA_KEY}",
        },
    )

    # Step-5: Instantiate and return "Rule" object, comprised of "variables", "domain_builder", "parameter_builders", and "expectation_configuration_builders" components.

+    variables: dict = {


This is the variable that has been introduced :) Thank you for this

Shinnnyshinshin · 2022-06-03T00:50:11Z

...sed_profiler/parameter_builder/mean_table_columns_set_match_multi_batch_parameter_builder.py

@@ -116,7 +116,7 @@ def _build_parameters(

        one_batch_table_columns_names_value: MetricValue
        multi_batch_table_columns_names_sets_as_list: List[Set[str]] = [
-            set(one_batch_table_columns_names_value.tolist())
+            set(one_batch_table_columns_names_value)


This was already a list :) bugfixed

Shinnnyshinshin · 2022-06-03T00:51:36Z

tests/cli/test_suite.py

-    semantic_types_dict=None,
-    table_expectations_only=False,
-    value_set_threshold="MANY",
+data_assistant_result: DataAssistantResult = context.assistants.onboarding.run(


This now includes all settings, including settings that are commented out. To be used as a starting point for documentation.

Shinnnyshinshin · 2022-06-03T00:52:09Z

tests/integration/docusaurus/tutorials/getting-started/getting_started.py

-    value_set_threshold="MANY",
-)
-suite = profiler.build_suite()
+data_assistant_result: DataAssistantResult = context.assistants.onboarding.run(


Similar story here. Including all settings even if they are not used.. to help documentation.

…https://github.com/great-expectations/great_expectations into feature/GREAT-953/migration-work-to-abc-data-context * 'feature/GREAT-953/migration-work-to-abc-data-context' of https://github.com/great-expectations/great_expectations: undo commit 2b0ca85 (#5253) Don't use jupyter-client 7.3.2 (#5252) Unpin cryptography upper bound (#5249) [DOCS] In the reference section of the ToC remove duplicates and update category pages (#5248) Fix Links (#5246) - updates formatting of class docs created by the auto API docs script to use bullet points for listing links to Public Methods. (#5247) [FEATURE] add new expectation on validating hexadecimals (#5188) [DOCS] Auto API documentation script (#4964) [BUGFIX] Hive temporary tables creation fix (#4956) [FEATURE] `DatasourceStore` (#5206) [FEATURE] Replace UserConfigurableProfiler with OnboardingDataAssistant in "CLI suite new --profile" Jupyter Notebooks (#5236)

Alex Sherstinsky added 3 commits June 2, 2022 16:37

Replace UserConfigurableProfiler with OnboardingDataAssistant in CLI …

a044cc2

…flows.

Merge branch 'develop' into feature/GREAT-761/GREAT-932/alexsherstins…

0dd172f

…ky/rule_based_profiler/replace_user_configurable_profiler_with_onboarding_data_assistant_in_cli_suite_new-2022_06_02-158

clean up

7e96a0c

alexsherstinsky enabled auto-merge (squash) June 3, 2022 00:06

alexsherstinsky requested review from cdkini, Shinnnyshinshin and NathanFarmer June 3, 2022 00:06

github-actions bot added the core-team label Jun 3, 2022

alexsherstinsky requested review from anthonyburdi, donaldheppner and talagluck June 3, 2022 00:06

type hints

6158875

alexsherstinsky disabled auto-merge June 3, 2022 00:14

alexsherstinsky enabled auto-merge (squash) June 3, 2022 00:14

alexsherstinsky disabled auto-merge June 3, 2022 00:34

Alex Sherstinsky added 2 commits June 2, 2022 17:35

clean up

727fe1f

clean up

cf9b1b7

alexsherstinsky enabled auto-merge (squash) June 3, 2022 00:44

alexsherstinsky disabled auto-merge June 3, 2022 00:51

Shinnnyshinshin approved these changes Jun 3, 2022

View reviewed changes

comments

c6643ed

alexsherstinsky enabled auto-merge (squash) June 3, 2022 01:10

alexsherstinsky disabled auto-merge June 3, 2022 01:47

Alex Sherstinsky added 2 commits June 2, 2022 22:29

clean up

0956972

clean up

50bd8e0

alexsherstinsky enabled auto-merge (squash) June 3, 2022 05:33

Alex Sherstinsky added 2 commits June 2, 2022 23:44

clean up tests

9891b3f

threading/pickling

3616e8f

alexsherstinsky disabled auto-merge June 3, 2022 06:51

alexsherstinsky enabled auto-merge (squash) June 3, 2022 06:51

alexsherstinsky merged commit 141dc2f into develop Jun 3, 2022

alexsherstinsky deleted the feature/GREAT-761/GREAT-932/alexsherstinsky/rule_based_profiler/replace_user_configurable_profiler_with_onboarding_data_assistant_in_cli_suite_new-2022_06_02-158 branch June 3, 2022 08:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] Replace UserConfigurableProfiler with OnboardingDataAssistant in "CLI suite new --profile" Jupyter Notebooks #5236

[FEATURE] Replace UserConfigurableProfiler with OnboardingDataAssistant in "CLI suite new --profile" Jupyter Notebooks #5236

alexsherstinsky commented Jun 3, 2022

netlify bot commented Jun 3, 2022 •

edited

Loading

ghost commented Jun 3, 2022 •

edited by ghost

Loading

Shinnnyshinshin left a comment

Shinnnyshinshin Jun 3, 2022

Shinnnyshinshin Jun 3, 2022

Shinnnyshinshin Jun 3, 2022

Shinnnyshinshin Jun 3, 2022

Shinnnyshinshin Jun 3, 2022

Shinnnyshinshin Jun 3, 2022

Shinnnyshinshin Jun 3, 2022

[FEATURE] Replace UserConfigurableProfiler with OnboardingDataAssistant in "CLI suite new --profile" Jupyter Notebooks #5236

[FEATURE] Replace UserConfigurableProfiler with OnboardingDataAssistant in "CLI suite new --profile" Jupyter Notebooks #5236

Conversation

alexsherstinsky commented Jun 3, 2022

Scope

Previous Design Review notes:

Definition of Done

netlify bot commented Jun 3, 2022 • edited Loading

✅ Deploy Preview for niobium-lead-7998 ready!

ghost commented Jun 3, 2022 • edited by ghost Loading

Legend

Shinnnyshinshin left a comment

Choose a reason for hiding this comment

Shinnnyshinshin Jun 3, 2022

Choose a reason for hiding this comment

Shinnnyshinshin Jun 3, 2022

Choose a reason for hiding this comment

Shinnnyshinshin Jun 3, 2022

Choose a reason for hiding this comment

Shinnnyshinshin Jun 3, 2022

Choose a reason for hiding this comment

Shinnnyshinshin Jun 3, 2022

Choose a reason for hiding this comment

Shinnnyshinshin Jun 3, 2022

Choose a reason for hiding this comment

Shinnnyshinshin Jun 3, 2022

Choose a reason for hiding this comment

netlify bot commented Jun 3, 2022 •

edited

Loading

ghost commented Jun 3, 2022 •

edited by ghost

Loading