[FEATURE] Implementing Fluent Datasources Support for Checkpoint #7697

alexsherstinsky · 2023-04-22T00:54:07Z

Scope

Regular Checkpoint now supports Fluent Datasources (including Usage Statistics)
Cleanup of typographical errors in Regular Checkpoint test module (for Block/V3 Datasources)

Next TODO

Verify SimpleCheckpoint supports for Fluent Datasources
Add tests for Checkpoint and SimpleCheckpoint utilizing Spark and SQL style Fluent Datasources.

Please annotate your PR title to describe what the PR does, then give a brief bulleted description of your PR below. PR titles should begin with [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], or [CONTRIB]. If a new feature introduces breaking changes for the Great Expectations API or configuration files, please also add [BREAKING]. You can read about the tags in our contributor checklist.

Changes proposed in this pull request:

JIRA: DX-469/DX-470

After submitting your PR, CI checks will run and @cla-bot will check for your CLA signature.

For a PR with nontrivial changes, we review with both design-centric and code-centric lenses.

In a design review, we aim to ensure that the PR is consistent with our relationship to the open source community, with our software architecture and abstractions, and with our users' needs and expectations. That review often starts well before a PR, for example in GitHub issues or Slack, so please link to relevant conversations in notes below to help reviewers understand and approve your PR more quickly (e.g. closes #123).

Previous Design Review notes:

Definition of Done

Please delete options that are not relevant.

My code follows the Great Expectations style guide
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added unit tests where applicable and made sure that new and existing tests are passing.
I have run any local integration tests and made sure that nothing is broken.

Thank you for submitting!

…/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_19-4

…ess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_19-4

…heckpoint_operation_with_fluent_datasource_batch_request-2023_04_19-4' into maintenance/DX-469/DX-470/alexsherstinsky/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-5

…sherstinsky/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_19-4' into maintenance/DX-469/DX-470/alexsherstinsky/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-5

…ure.

…ethod names.

…/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-6

…heckpoint_operation_with_fluent_datasource_batch_request-2023_04_20-6' into maintenance/DX-469/DX-470/alexsherstinsky/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-7

netlify · 2023-04-22T00:54:12Z

✅ Deploy Preview for niobium-lead-7998 canceled.

Name	Link
🔨 Latest commit	`22c4eee`
🔍 Latest deploy log	https://app.netlify.com/sites/niobium-lead-7998/deploys/64443f9e59280f000880362d

alexsherstinsky · 2023-04-22T00:56:14Z

great_expectations/datasource/fluent/config.py

@@ -349,7 +349,7 @@ def _convert_fluent_datasources_loaded_from_yaml_to_internal_object_representati
    config: Dict[str, Any], _allow_empty: bool = False
 ) -> Dict[str, Any]:
    if _FLUENT_DATASOURCES_KEY in config:
-        fluent_datasources: dict = config[_FLUENT_DATASOURCES_KEY]
+        fluent_datasources: dict = config[_FLUENT_DATASOURCES_KEY] or {}


@Kilo59 This is needed, because great_expectations.yml may come with an empty fluent_datasources section.

ghost · 2023-04-22T00:56:38Z

👇 Click on the image for a new way to code review

Legend

alexsherstinsky · 2023-04-22T00:57:42Z

great_expectations/core/usage_statistics/anonymizers/types/base.py

@@ -15,6 +15,7 @@
 BATCH_REQUEST_OPTIONAL_TOP_LEVEL_KEYS: Set[str] = {
    "data_connector_query",
    "runtime_parameters",
+    "options",


This allows Usage Statistics to work with Checkpoints using Fluent Datasources out of the box.

alexsherstinsky · 2023-04-22T00:57:49Z

great_expectations/datasource/fluent/file_path_data_asset.py

@@ -59,6 +59,7 @@ class _FilePathDataAsset(DataAsset):
        "batching_regex",  # file_path argument
        "kwargs",  # kwargs need to be unpacked and passed separately
        "batch_metadata",
+        "connect_options",


@Kilo59 This is needed -- otherwise, instantiating an ExecutionEngine says that "options is not recognized by init().

alexsherstinsky · 2023-04-22T00:58:29Z

tests/checkpoint/conftest.py

@@ -25,6 +25,36 @@ def titanic_pandas_data_context_stats_enabled_and_expectation_suite_with_one_exp
    return context


+@pytest.fixture


We might embellish this fixture later (e.g., if additional Fluent Datasources and/or DataAssets are needed).

alexsherstinsky · 2023-04-22T00:59:09Z

tests/checkpoint/test_checkpoint.py

@@ -127,7 +127,7 @@ def test_basic_checkpoint_config_validation(
    caplog,
    capsys,
 ):
-    context: FileDataContext = empty_data_context_stats_enabled
+    data_context: FileDataContext = empty_data_context_stats_enabled


These changes are only beautification focused and correcting typographical errors. No logic has been changed.

alexsherstinsky · 2023-04-22T00:59:56Z

tests/checkpoint/test_checkpoint_with_fluent_datasources.py

@@ -0,0 +1,1164 @@
+import logging


We should also add tests for Checkpoint and SimpleCheckpoint utilizing Spark and SQL style Fluent Datasources.

…ore" macros.

…ess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-7

…/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-7

…sherstinsky/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-7' into maintenance/DX-469/DX-470/alexsherstinsky/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-7

…/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-7

lukedyer-peak · 2023-05-02T08:29:29Z

great_expectations/checkpoint/checkpoint.py

-            # Next two fields are for LegacyCheckpoint configuration
-            "validation_operator_name": validation_operator_name,
-            "batches": batches,
-            # the following four keys are used by SimpleCheckpoint
-            "site_names": site_names,
-            "slack_webhook": slack_webhook,
-            "notify_on": notify_on,
-            "notify_with": notify_with,


I've subclassed a SimpleCheckpoint and this change has broken inheritance

alexsherstinsky · 2023-05-02T15:44:53Z

@lukedyer-peak Thank you very much for reporting this. Is there any chance for you to update your class without too much difficulty? If so, that would be of great help; otherwise, please do let me know as soon as possible, and I will try to figure out a solution on our side. Again, apologies that this change broke your workflow, and please let me know. Thanks again.

lukedyer-peak · 2023-05-16T08:07:03Z

Note: conversation moved to #7790

Alex Sherstinsky and others added 20 commits April 20, 2023 09:18

Minor fixes to Checkpoint implementation and tests.

7924868

Minor fixes to Checkpoint implementation and tests.

8940e6f

Merge branch 'develop' into maintenance/DX-469/DX-470/alexsherstinsky…

66ab073

…/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_19-4

Minor fixes to Checkpoint implementation and tests.

2a2ddef

Merge develop into maintenance/DX-469/DX-470/alexsherstinsky/link/ass…

35fcad3

…ess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_19-4

Merge branch 'maintenance/DX-469/DX-470/alexsherstinsky/link/assess_c…

323fa89

…heckpoint_operation_with_fluent_datasource_batch_request-2023_04_19-4' into maintenance/DX-469/DX-470/alexsherstinsky/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-5

Clean up checkpoint test method names and usage of batch_request fixt…

47a6809

…ure.

New fixtures for Fluent Datasources and correcting spelling fixture m…

b84bb7c

…ethod names.

Merge branch 'develop' into maintenance/DX-469/DX-470/alexsherstinsky…

7167c03

…/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-6

clean up

32253a4

WIP

51c8712

clean up

4d25f83

clean up

a4d61c1

Merge branch 'develop' into maintenance/DX-469/DX-470/alexsherstinsky…

f769717

…/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-6

Merge branch 'maintenance/DX-469/DX-470/alexsherstinsky/link/assess_c…

56363ac

…heckpoint_operation_with_fluent_datasource_batch_request-2023_04_20-6' into maintenance/DX-469/DX-470/alexsherstinsky/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-7

WIP

c9827b9

Supporting Fluent Datasources for Checkpoint (regular Checkpoint).

a2376da

Supporting Fluent Datasources for Checkpoint (regular Checkpoint).

88fe336

merge

8fb6dc2

github-actions bot added core dx labels Apr 22, 2023

alexsherstinsky commented Apr 22, 2023

View reviewed changes

Alex Sherstinsky added 2 commits April 21, 2023 18:23

Thanks to utilization of "FluentBatchRequest", we now need fewer "ign…

375c62e

…ore" macros.

Thanks to utilization of "FluentBatchRequest", we now need fewer "ign…

0c5f069

…ore" macros.

alexsherstinsky marked this pull request as ready for review April 22, 2023 01:36

alexsherstinsky enabled auto-merge (squash) April 22, 2023 01:41

alexsherstinsky requested review from a team April 22, 2023 01:42

github-actions bot and others added 3 commits April 22, 2023 14:42

Merge develop into maintenance/DX-469/DX-470/alexsherstinsky/link/ass…

0f4ecec

…ess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-7

reverting rename of context to data_context

82f3da4

Merge develop into maintenance/DX-469/DX-470/alexsherstinsky/link/ass…

36d1b2e

…ess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-7

alexsherstinsky disabled auto-merge April 22, 2023 16:50

donaldheppner and others added 8 commits April 22, 2023 13:27

removed "options" from config and dicts for fluent batch requests

01fa53b

Merge branch 'develop' into maintenance/DX-469/DX-470/alexsherstinsky…

b2487b8

…/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-7

blacked

23245ce

clean up

1c3ff4d

clean up

8ad5d96

Merge branch 'develop' into maintenance/DX-469/DX-470/alexsherstinsky…

68dfab4

…/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-7

alexsherstinsky enabled auto-merge (squash) April 22, 2023 19:44

Alex Sherstinsky added 2 commits April 22, 2023 12:50

Merge branch 'develop' into maintenance/DX-469/DX-470/alexsherstinsky…

29cf3c0

…/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-7

Supporting Fluent Datasources for Checkpoint (regular Checkpoint).

22c4eee

donaldheppner approved these changes Apr 22, 2023

View reviewed changes

alexsherstinsky force-pushed the maintenance/DX-469/DX-470/alexsherstinsky/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-7 branch from 4ce73ee to 22c4eee Compare April 22, 2023 20:12

alexsherstinsky merged commit 3fe2ec4 into develop Apr 22, 2023

alexsherstinsky deleted the maintenance/DX-469/DX-470/alexsherstinsky/link/assess_checkpoint_operation_with_fluent_datasource_batch_request-2023_04_20-7 branch April 22, 2023 21:09

dctalbot pushed a commit that referenced this pull request Apr 24, 2023

[FEATURE] Implementing Fluent Datasources Support for Checkpoint (#7697)

d5f09e7

lukedyer-peak reviewed May 2, 2023

View reviewed changes

lukedyer-peak mentioned this pull request May 2, 2023

Broken inheritance from PR "Fluent datasource support for checkpoints" #7697 #7790

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] Implementing Fluent Datasources Support for Checkpoint #7697

[FEATURE] Implementing Fluent Datasources Support for Checkpoint #7697

alexsherstinsky commented Apr 22, 2023 •

edited

Loading

netlify bot commented Apr 22, 2023 •

edited

Loading

alexsherstinsky Apr 22, 2023

ghost commented Apr 22, 2023 •

edited by ghost

Loading

Legend

alexsherstinsky Apr 22, 2023

alexsherstinsky Apr 22, 2023

alexsherstinsky Apr 22, 2023

alexsherstinsky Apr 22, 2023

alexsherstinsky Apr 22, 2023 •

edited

Loading

lukedyer-peak May 2, 2023

alexsherstinsky commented May 2, 2023

lukedyer-peak commented May 16, 2023

		@@ -25,6 +25,36 @@ def titanic_pandas_data_context_stats_enabled_and_expectation_suite_with_one_exp
		return context


		@pytest.fixture

[FEATURE] Implementing Fluent Datasources Support for Checkpoint #7697

[FEATURE] Implementing Fluent Datasources Support for Checkpoint #7697

Conversation

alexsherstinsky commented Apr 22, 2023 • edited Loading

Scope

Next TODO

Previous Design Review notes:

Definition of Done

netlify bot commented Apr 22, 2023 • edited Loading

✅ Deploy Preview for niobium-lead-7998 canceled.

alexsherstinsky Apr 22, 2023

Choose a reason for hiding this comment

ghost commented Apr 22, 2023 • edited by ghost Loading

Legend

alexsherstinsky Apr 22, 2023

Choose a reason for hiding this comment

alexsherstinsky Apr 22, 2023

Choose a reason for hiding this comment

alexsherstinsky Apr 22, 2023

Choose a reason for hiding this comment

alexsherstinsky Apr 22, 2023

Choose a reason for hiding this comment

alexsherstinsky Apr 22, 2023 • edited Loading

Choose a reason for hiding this comment

lukedyer-peak May 2, 2023

Choose a reason for hiding this comment

alexsherstinsky commented May 2, 2023

lukedyer-peak commented May 16, 2023

alexsherstinsky commented Apr 22, 2023 •

edited

Loading

netlify bot commented Apr 22, 2023 •

edited

Loading

ghost commented Apr 22, 2023 •

edited by ghost

Loading

alexsherstinsky Apr 22, 2023 •

edited

Loading