[BUGFIX] Fix copying of CheckpointConfig for substitution and printing purposes #3759

alexsherstinsky · 2021-11-30T00:59:31Z

Scope

This work fixes a bug in CheckpointConfig serialization, specifically the loss of batch_data as the side effect.

The implementation consists of:

Overriding CheckpointConfigSchema.dump(),
Overriding CheckpointConfig.__deepcopy__(), and
Implementing CheckpointConfig.__repr__() and CheckpointResult.__repr__() methods

in the way that:

Extracts the batch_data values from the CheckpointConfiguration and saving them.
Deleting the batch_data values from the CheckpointConfiguration and cloning/serializing the resulting "safe" configuration object.
Restoring the batch_data values into the cloned and/or original configuration object.

Additional clean up consists of:

Fixing test(s) that returned incorrect results due to the previous implementation.
Making the get_runtime_batch_request() Checkpoint utility more elegant.

As the result of the above fixes, the impacted modules became simpler, and the amount of code has been reduced.

Please annotate your PR title to describe what the PR does, then give a brief bulleted description of your PR below. PR titles should begin with [BUGFIX], [FEATURE], [DOCS], or [MAINTENANCE]. If a new feature introduces breaking changes for the Great Expectations API or configuration files, please also add [BREAKING]. You can read about the tags in our contributor checklist.

Changes proposed in this pull request:

JIRA: ANT-20/ANT-22/ANT-57

After submitting your PR, CI checks will run and @tiny-tim-bot will check for your CLA signature.

For a PR with nontrivial changes, we review with both design-centric and code-centric lenses.

In a design review, we aim to ensure that the PR is consistent with our relationship to the open source community, with our software architecture and abstractions, and with our users' needs and expectations. That review often starts well before a PR, for example in github issues or slack, so please link to relevant conversations in notes below to help reviewers understand and approve your PR more quickly (e.g. closes #123).

Previous Design Review notes:

Definition of Done

Please delete options that are not relevant.

My code follows the Great Expectations style guide
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added unit tests where applicable and made sure that new and existing tests are passing.
I have run any local integration tests and made sure that nothing is broken.

Thank you for submitting!

netlify · 2021-11-30T00:59:36Z

✔️ Deploy Preview for niobium-lead-7998 ready!

🔨 Explore the source changes: e769faa

🔍 Inspect the deploy log: https://app.netlify.com/sites/niobium-lead-7998/deploys/61a7a481dd5b66000704eb93

😎 Browse the preview: https://deploy-preview-3759--niobium-lead-7998.netlify.app

github-actions · 2021-11-30T00:59:52Z

HOWDY! This is your friendly 🤖 CHANGELOG bot 🤖

Please don't forget to add a clear and succinct description of your change under the Develop header in docs_rtd/changelog.rst, if applicable. This will help us with the release process. See the Contribution checklist in the Great Expectations documentation for the type of labels to use!

✨ Thank you! ✨

donaldheppner · 2021-11-30T12:59:19Z

great_expectations/checkpoint/checkpoint.py

@@ -377,13 +327,13 @@ def run(
            "template_name": template_name,
            "run_name_template": run_name_template,
            "expectation_suite_name": expectation_suite_name,
-            "expectation_suite_ge_cloud_id": expectation_suite_ge_cloud_id,


why was this line removed?

Looks like it was just moved to the bottom (new line 336)

@donaldheppner I simply moved it to a "more logical place" congruent with the order these arguments were supplied on input to the method.

donaldheppner · 2021-11-30T13:02:48Z

great_expectations/checkpoint/types/checkpoint_result.py

@@ -11,6 +10,9 @@
 from great_expectations.data_context.types.base import (
    CheckpointConfig,
    CheckpointConfigSchema,
+    delete_runtime_parameters_batch_data_references_from_config,


could/should these not be methods on the config object?

@donaldheppner In fact, even better: in the next commit, they will be moved to batch.py core module.

Thank you @alexsherstinsky, if that is the case, could I ask that individual tests for the methods be added in the next commit (or is it PR?)

great_expectations/checkpoint/types/checkpoint_result.py

donaldheppner · 2021-11-30T13:06:31Z

great_expectations/checkpoint/util.py

@@ -207,26 +207,11 @@ def send_email(
 # TODO: <Alex>A common utility function should be factored out from DataContext.get_batch_list() for any purpose.</Alex>


Is this TODO still valid?

@donaldheppner Unfortunately, this comment is still valid -- we have not had a chance to do the work. I would like to leave it for now and remove only after we had a chance to go through the backlog formally. Thank you.

NathanFarmer

Just a few questions!

NathanFarmer · 2021-11-30T13:25:13Z

great_expectations/data_context/types/base.py

-        if (batch_data is not None) and (
-            self.batch_request.get("runtime_parameters") is not None
+
+def delete_runtime_parameters_batch_data_references_from_config(


NathanFarmer · 2021-11-30T13:25:35Z

great_expectations/data_context/types/base.py

+                    val["batch_request"]["runtime_parameters"].pop("batch_data")
+
+
+def restore_runtime_parameters_batch_data_references_into_config(


NathanFarmer · 2021-11-30T13:32:00Z

great_expectations/data_context/types/base.py

+    def __deepcopy__(self, memo):
+        batch_data_references: Tuple[
+            Optional[Any], Optional[List[Any]]
+        ] = get_runtime_parameters_batch_data_references_from_config(config=self)
+        delete_runtime_parameters_batch_data_references_from_config(config=self)
+
+        cls = self.__class__
+        result = cls.__new__(cls)
+        result._commented_map = CommentedMap()
+
+        memo[id(self)] = result
+        for key, value in self.to_json_dict().items():
+            # noinspection PyArgumentList
+            value_copy = copy.deepcopy(value, memo)
+            setattr(result, key, value_copy)
+
+        restore_runtime_parameters_batch_data_references_into_config(
+            config=self, batch_data_references=batch_data_references
+        )
+        restore_runtime_parameters_batch_data_references_into_config(
+            config=result, batch_data_references=batch_data_references
+        )
+
+        return result


Can a custom __deepcopy__ not be avoided here? Why isn't this something we can do where required, like we are doing in other places using these new functions in this PR?

@NathanFarmer This custom __deepcoppy__ enables us to simplify / delete a lot of code. It fits, because it "knows" how to handle the copying for the (complex) CheckpointConfig object.

NathanFarmer · 2021-11-30T13:34:15Z

tests/checkpoint/test_checkpoint.py

@@ -2337,6 +2337,7 @@ def test_newstyle_checkpoint_config_substitution_nested(
        CheckpointConfig(
            name="my_nested_checkpoint",
            config_version=1,
+            template_name="my_nested_checkpoint_template_3",


Is there a new test to be added for the bug you referenced?

@NathanFarmer This test has effectively been added by repairing some existing tests. In addition, the functionality will be fully manifested in a near-future PR that focuses on checkpoint.run payload instrumentation, which is how the bug was detected in the first place (the batch_data values were empty in the runtime_parameter dictionary). Thanks.

NathanFarmer · 2021-11-30T13:36:12Z

great_expectations/checkpoint/types/checkpoint_result.py

-
-        return json_dict
+    def to_json_dict(self) -> dict:
+        return checkpointResultSchema.dump(self)


anthonyburdi

Looks good to me aside from resolving Don and Nathan's comments :)

anthonyburdi · 2021-11-30T17:14:30Z

great_expectations/checkpoint/checkpoint.py

@@ -377,13 +327,13 @@ def run(
            "template_name": template_name,
            "run_name_template": run_name_template,
            "expectation_suite_name": expectation_suite_name,
-            "expectation_suite_ge_cloud_id": expectation_suite_ge_cloud_id,


Looks like it was just moved to the bottom (new line 336)

Shinnnyshinshin

Thank you very much for this work @alexsherstinsky. Looks good on my end too, with one clarification question

Shinnnyshinshin · 2021-11-30T17:36:24Z

great_expectations/core/batch.py

@@ -361,8 +361,8 @@ def _validate_runtime_batch_request_specific_init_parameters(
    ):
        if not (runtime_parameters and (isinstance(runtime_parameters, dict))):
            raise TypeError(
-                f"""The type for runtime_parameters must be a dict object.
-                The type given is "{str(type(runtime_parameters))}", which is illegal."""
+                f"""The runtime_parameters must be a non-empty dict object.


Very helpful error message. Thank you for this

Shinnnyshinshin · 2021-11-30T17:42:24Z

great_expectations/data_context/types/base.py

+                    val["batch_request"]["runtime_parameters"].pop("batch_data")
+
+
+def restore_runtime_parameters_batch_data_references_into_config(


Shinnnyshinshin · 2021-11-30T17:42:34Z

great_expectations/data_context/types/base.py

-        if (batch_data is not None) and (
-            self.batch_request.get("runtime_parameters") is not None
+
+def delete_runtime_parameters_batch_data_references_from_config(


Shinnnyshinshin · 2021-11-30T17:44:13Z

great_expectations/checkpoint/types/checkpoint_result.py

@@ -11,6 +10,9 @@
 from great_expectations.data_context.types.base import (
    CheckpointConfig,
    CheckpointConfigSchema,
+    delete_runtime_parameters_batch_data_references_from_config,


Thank you @alexsherstinsky, if that is the case, could I ask that individual tests for the methods be added in the next commit (or is it PR?)

roblim · 2021-11-30T23:44:04Z

great_expectations/checkpoint/util.py

+    effective_batch_request: dict = dict(
+        **runtime_config_batch_request, **validation_batch_request
+    )
+    if "runtime_parameters" in effective_batch_request or isinstance(
+        validation_batch_request, RuntimeBatchRequest
+    ):
+        batch_request_class = RuntimeBatchRequest
+    else:
+        batch_request_class = BatchRequest
+


Much better!

…NT-22/alexsherstinsky/instrument_runtime_data_connector_for_usage_statistics-2021_11_23-10-overload_deepcopy_and_dump-2021_11_29-11

alexsherstinsky · 2021-12-01T06:11:56Z

Tests have been added for new utilities.

In addition, BatchRequest serialization has also been dramatically simplified using these new utilities.

NathanFarmer

It is a thing of beauty. Just one question about a TODO.

NathanFarmer · 2021-12-01T14:11:27Z

great_expectations/data_context/types/base.py

+            config=data, batch_data_references=batch_data_references
+        )
+        return data
+

 class CheckpointConfig(BaseYamlConfig):
    # TODO: <Alex>ALEX (does not work yet)</Alex>


Can we remove this?

@NathanFarmer We still need those comments -- the idea was good, but it did not work, and we ought to figure out how to make it work -- or discuss it and conclude that the current approach is the best that can be done. Thanks!

…trument_runtime_data_connector_for_usage_statistics-2021_11_23-10-overload_deepcopy_and_dump-2021_11_29-11

NathanFarmer

This is a wonderful refactor 😄 LGTM!

Shinnnyshinshin

Wonderful work 🙇🏼 thank you @alexsherstinsky

roblim

Looks great! <3

anthonyburdi

LGTM!

anthonyburdi · 2021-12-01T17:42:31Z

great_expectations/core/batch.py

+                    ] = validations_batch_data_list[idx]
+
+
+def standardize_batch_request_display_ordering(


anthonyburdi · 2021-12-01T17:43:25Z

great_expectations/core/usage_statistics/anonymizers/batch_request_anonymizer.py

@@ -40,6 +41,9 @@ def anonymize_batch_request(self, *args, **kwargs) -> Dict[str, List[str]]:
        anonymized_batch_request_dict: Optional[
            Union[str, dict]
        ] = self._anonymize_batch_request_properties(source=batch_request_dict)
+        anonymized_batch_request_dict = standardize_batch_request_display_ordering(


anthonyburdi · 2021-12-01T17:53:35Z

tests/core/test_serialization.py

@@ -59,3 +74,771 @@ def test_serialization_of_spark_df(spark_session):
    df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
    sdf = spark_session.createDataFrame(df)
    assert convert_to_json_serializable(sdf) == {"a": [1, 2, 3], "b": [4, 5, 6]}
+
+
+def test_batch_request_deepcopy():


Thank you for adding these tests!

Alex Sherstinsky added 7 commits November 29, 2021 12:19

SAVE-works-WIP

83b3e2d

simplify copying checkpoint configuration

8ae9055

debug version

8f4860f

clean up

c1768f5

Linting

bbe0734

making code more elegant

3843b93

warnings wording

22c1fc9

Alex Sherstinsky added 2 commits November 29, 2021 17:25

Fix test broken due to previous implementation.

ba6d76d

clean up

11141e8

alexsherstinsky marked this pull request as ready for review November 30, 2021 02:41

alexsherstinsky requested review from NathanFarmer, anthonyburdi, roblim, donaldheppner, jcampbell and Shinnnyshinshin November 30, 2021 02:41

donaldheppner reviewed Nov 30, 2021

View reviewed changes

NathanFarmer requested changes Nov 30, 2021

View reviewed changes

anthonyburdi reviewed Nov 30, 2021

View reviewed changes

Shinnnyshinshin reviewed Nov 30, 2021

View reviewed changes

alexsherstinsky requested review from cdkini and talagluck November 30, 2021 22:39

roblim reviewed Nov 30, 2021

View reviewed changes

Alex Sherstinsky added 4 commits November 30, 2021 20:21

further efficiencies

a2b17d3

Merge remote-tracking branch 'upstream/develop' into feature/ANT-20/A…

ae7b7ff

…NT-22/alexsherstinsky/instrument_runtime_data_connector_for_usage_statistics-2021_11_23-10-overload_deepcopy_and_dump-2021_11_29-11

Further coding efficiencies and elegance.

2d3b8ab

cleanup

4f87a42

alexsherstinsky requested review from roblim, anthonyburdi, donaldheppner, NathanFarmer and Shinnnyshinshin December 1, 2021 06:12

NathanFarmer reviewed Dec 1, 2021

View reviewed changes

Merge branch 'develop' into feature/ANT-20/ANT-22/alexsherstinsky/ins…

de7db46

…trument_runtime_data_connector_for_usage_statistics-2021_11_23-10-overload_deepcopy_and_dump-2021_11_29-11

alexsherstinsky requested a review from NathanFarmer December 1, 2021 16:33

e769faa

NathanFarmer approved these changes Dec 1, 2021

View reviewed changes

Shinnnyshinshin approved these changes Dec 1, 2021

View reviewed changes

alexsherstinsky merged commit 954da9f into develop Dec 1, 2021

alexsherstinsky deleted the feature/ANT-20/ANT-22/alexsherstinsky/instrument_runtime_data_connector_for_usage_statistics-2021_11_23-10-overload_deepcopy_and_dump-2021_11_29-11 branch December 1, 2021 17:27

roblim reviewed Dec 1, 2021

View reviewed changes

anthonyburdi approved these changes Dec 1, 2021

View reviewed changes

fjork3 mentioned this pull request Dec 2, 2021

release candidate for 0.13.45 #3781

Merged

		@@ -207,26 +207,11 @@ def send_email(
		# TODO: <Alex>A common utility function should be factored out from DataContext.get_batch_list() for any purpose.</Alex>

		val["batch_request"]["runtime_parameters"].pop("batch_data")


		def restore_runtime_parameters_batch_data_references_into_config(

		] = validations_batch_data_list[idx]


		def standardize_batch_request_display_ordering(

[BUGFIX] Fix copying of CheckpointConfig for substitution and printing purposes #3759

[BUGFIX] Fix copying of CheckpointConfig for substitution and printing purposes #3759

Conversation

alexsherstinsky commented Nov 30, 2021 • edited

Scope

Previous Design Review notes:

Definition of Done

netlify bot commented Nov 30, 2021 • edited

github-actions bot commented Nov 30, 2021

HOWDY! This is your friendly 🤖 CHANGELOG bot 🤖

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NathanFarmer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anthonyburdi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Shinnnyshinshin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexsherstinsky commented Dec 1, 2021

NathanFarmer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NathanFarmer left a comment

Choose a reason for hiding this comment

Shinnnyshinshin left a comment

Choose a reason for hiding this comment

roblim left a comment

Choose a reason for hiding this comment

anthonyburdi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexsherstinsky commented Nov 30, 2021 •

edited

netlify bot commented Nov 30, 2021 •

edited