[MAINTENANCE] Clean up public api report part 3 #6803

anthonyburdi · 2023-01-13T17:28:13Z

Changes proposed in this pull request:

Add excludes & includes to the public api report from invoke public-api
This is part 3, which completes the first pass of the items in the public api report.
To see the full list, run invoke public-api, or look at the tracker with rationales.

This PR builds on #6784 and #6792.

Definition of Done

My code follows the Great Expectations style guide
I have performed a self-review of my own code

ghost · 2023-01-13T17:29:11Z

👇 Click on the image for a new way to code review

Make big changes easier — review code in small groups of related files
Know where to start — see the whole change at a glance
Take a code tour — explore the change with an interactive tour
Make comments and review — all fully sync’ed with github

Try it now!

Legend

netlify · 2023-01-13T17:29:37Z

✅ Deploy Preview for niobium-lead-7998 ready!

Name	Link
🔨 Latest commit	`5be5c70`
🔍 Latest deploy log	https://app.netlify.com/sites/niobium-lead-7998/deploys/63c6c25ff734d7000815dcdf
😎 Deploy Preview	https://deploy-preview-6803--niobium-lead-7998.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

talagluck

Looks good overall! A couple of questions and some opinion base changes.

scripts/public_api_report.py

talagluck · 2023-01-17T14:04:03Z

scripts/public_api_report.py

+        IncludeExcludeDefinition(
+            reason="False match for context.get_validator()",
+            name="get_validator",
+            filepath=pathlib.Path(
+                "great_expectations/rule_based_profiler/parameter_builder/parameter_builder.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="Not used directly but from Data Assistant or RuleBasedProfiler",
+            name="run",
+            filepath=pathlib.Path(
+                "great_expectations/rule_based_profiler/rule/rule.py"
+            ),
+        ),


Are these needed for the Data Assistant contribution process?

I don't see either Rule.run() or ParameterBuilder.get_validator() used in DataAssistant, OnboardingDataAssistant or the VolumeDataAssistant. It looks like it gets called internally.
Could you point me to the Data Assistant contribution process to double check? I haven't been able to find it in our docs.

Ah, it doesn't exist yet! Though it is on the roadmap. Perhaps @alexsherstinsky can help answer this? Otherwise we can always add this to the Public API when it's needed.

talagluck · 2023-01-17T14:05:41Z

scripts/public_api_report.py

+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/checkpoint/types/checkpoint_result.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path("great_expectations/core/batch.py"),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path("great_expectations/core/domain.py"),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/core/expectation_configuration.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/core/expectation_diagnostics/expectation_diagnostics.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path("great_expectations/core/expectation_suite.py"),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/core/expectation_validation_result.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="Internal use",
+            name="IDDict",
+            filepath=pathlib.Path("great_expectations/core/id_dict.py"),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path("great_expectations/core/run_identifier.py"),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path("great_expectations/data_context/types/base.py"),
+        ),


I think these should be public. They are used for passing things around in the Airflow Operator, and some users use it when they need to pass around various forms of config. My hope is that the need for it goes away when we transition to New Style Datasources, but we’re not quite there yet.

Updated (not all, see 5be5c70 for specifics)

talagluck · 2023-01-17T14:07:28Z

scripts/public_api_report.py

+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/rule_based_profiler/attributed_resolved_metrics.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path("great_expectations/rule_based_profiler/builder.py"),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/rule_based_profiler/config/base.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/rule_based_profiler/data_assistant_result/data_assistant_result.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/rule_based_profiler/estimators/numeric_range_estimation_result.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/rule_based_profiler/estimators/numeric_range_estimator.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/rule_based_profiler/helpers/cardinality_checker.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/rule_based_profiler/helpers/configuration_reconciliation.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/rule_based_profiler/helpers/runtime_environment.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/rule_based_profiler/parameter_container.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/rule_based_profiler/rule/rule.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/rule_based_profiler/rule_based_profiler.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/rule_based_profiler/rule_based_profiler_result.py"
+            ),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path("great_expectations/types/attributes.py"),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path("great_expectations/types/base.py"),
+        ),


See above. I don't think all to_json_dict methods need to be public, but I think any classes that are public, and/or whose config objects are public, should have public to_json_dict methods.

Updated (see 5be5c70 for specifics)

talagluck · 2023-01-17T14:07:39Z

scripts/public_api_report.py

+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path("great_expectations/validator/exception_info.py"),
+        ),
+        IncludeExcludeDefinition(
+            reason="to_json_dict is an internal helper method",
+            name="to_json_dict",
+            filepath=pathlib.Path(
+                "great_expectations/validator/metric_configuration.py"
+            ),
+        ),


Updated (not all, see 5be5c70 for specifics)

talagluck

LGTM! Thanks for the updates. For the RBP stuff, we can always add this to the Public API when we're ready for the DA contribution docs, so no need to gate on that.

…ttps://github.com/great-expectations/great_expectations into m/dx-21/dx-157/suppress-warnings-for-result-format * 'm/dx-21/dx-157/suppress-warnings-for-result-format' of https://github.com/great-expectations/great_expectations: (57 commits) [MAINTENANCE] Use shortened_dotted_paths in API docs (#6820) [MAINTENANCE] Use Enum classes for all metric name suffixes (#6819) [BUGFIX]: expect_day_count_to_be_close_to_equivalent_week_day_mean (#6811) [CONTRIB] Improve contrib schwifty expectations (#6812) Add docstring for `AbstractDataContext.add_checkpoint` (#6728) [BUGFIX] Implemented lowercase function to check what type of file endswith (#6810) [DOCS] ExpectationSuite and remove_expectation api docs (#6785) [MAINTENANCE] Clean up public api report part 3 (#6803) [MAINTENANCE] Fixes for column_values_to_be_between tests (#6809) [MAINTENANCE] Improve type hints in ExecutionEngine.resolve_metrics() flow and delete unnecessary checks (#6804) [BUGFIX] Fix missing not operator ~ (#6808) Suppress 2 kl_divergence datasets for bigquery that took 90 minutes to insert (#6807) [MAINTENANCE] Use environment variables for expectation gallery data paths (#6805) [MAINTENANCE] Add checkpoint and datadoc integration test for zep pandas datasource. (#6793) [DOCS] Link to gh issue #4152 for ruamel.yaml (#6799) [MAINTENANCE] Misc docstrings in `AbstractDataContext` (#6801) [MAINTENANCE] Shift daily gallery run by 6 hours (#6802) [MAINTENANCE] Clean up public api report part 2 (#6792) [MAINTENANCE] Clean up public api report part 1 (#6784) [MAINTENANCE] Only include relevant diagnostics info in gallery JSON (#6797) ...

anthonyburdi added 2 commits January 13, 2023 12:16

Include ValidationActions

2f41cfe

Exclude items

c964542

github-actions bot added core-team dx labels Jan 13, 2023

anthonyburdi requested review from a team and talagluck January 13, 2023 17:29

anthonyburdi self-assigned this Jan 13, 2023

anthonyburdi mentioned this pull request Jan 13, 2023

[MAINTENANCE] Clean up public api report part 2 #6792

Merged

2 tasks

anthonyburdi added 2 commits January 13, 2023 17:32

Merge branch 'develop' into m/dx-188/clean_up_public_api_report_part_3

4fe8885

Update list from recent run

a52d128

talagluck reviewed Jan 17, 2023

View reviewed changes

anthonyburdi added 2 commits January 17, 2023 10:15

Merge branch 'develop' into m/dx-188/clean_up_public_api_report_part_3

2d51932

Update per Tal's comments

5be5c70

talagluck approved these changes Jan 17, 2023

View reviewed changes

anthonyburdi enabled auto-merge (squash) January 17, 2023 16:13

anthonyburdi merged commit a89d52e into develop Jan 17, 2023

anthonyburdi deleted the m/dx-188/clean_up_public_api_report_part_3 branch January 17, 2023 16:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MAINTENANCE] Clean up public api report part 3 #6803

[MAINTENANCE] Clean up public api report part 3 #6803

anthonyburdi commented Jan 13, 2023 •

edited

Loading

ghost commented Jan 13, 2023 •

edited by ghost

Loading

netlify bot commented Jan 13, 2023 •

edited

Loading

talagluck left a comment

talagluck Jan 17, 2023

anthonyburdi Jan 17, 2023

talagluck Jan 17, 2023

talagluck Jan 17, 2023

anthonyburdi Jan 17, 2023

talagluck Jan 17, 2023

anthonyburdi Jan 17, 2023

talagluck Jan 17, 2023

anthonyburdi Jan 17, 2023

talagluck left a comment

[MAINTENANCE] Clean up public api report part 3 #6803

[MAINTENANCE] Clean up public api report part 3 #6803

Conversation

anthonyburdi commented Jan 13, 2023 • edited Loading

Definition of Done

ghost commented Jan 13, 2023 • edited by ghost Loading

Legend

netlify bot commented Jan 13, 2023 • edited Loading

✅ Deploy Preview for niobium-lead-7998 ready!

talagluck left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

talagluck left a comment

Choose a reason for hiding this comment

anthonyburdi commented Jan 13, 2023 •

edited

Loading

ghost commented Jan 13, 2023 •

edited by ghost

Loading

netlify bot commented Jan 13, 2023 •

edited

Loading