Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUGFIX] Ensure that Checkpoint deserializes proper action subclass #9701

Merged
merged 4 commits into from Apr 3, 2024

Conversation

cdkini
Copy link
Member

@cdkini cdkini commented Apr 3, 2024

Without the specific union noted in this PR, we get a generic ValidationAction type when reading from disk.

  • Description of PR changes above includes a link to an existing GitHub issue
  • PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB]
  • Code is linted - run invoke lint (uses ruff format + ruff check)
  • Appropriate tests and docs have been updated

For more information about contributing, see Contribute.

After you submit your PR, keep the page open and monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up and reach out on Slack if you need help. Thanks for contributing!

Copy link

netlify bot commented Apr 3, 2024

Deploy Preview for niobium-lead-7998 canceled.

Name Link
🔨 Latest commit 08247f3
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/660dbb82b0e3710008b7713a

ids=[k.__name__ for k in ACTION_INIT_PARAMS],
)
@pytest.mark.unit
def test_action_deserialization_within_parent_class(
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we can test this with Checkpoint, I'd like to remove this test

@cdkini cdkini self-assigned this Apr 3, 2024
def test_checkpoint_deserialization_with_actions(self, mocker: MockerFixture):
# Arrange
context = mocker.Mock(spec=AbstractDataContext)
context.validation_definition_store.get.return_value = mocker.Mock(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deserializing a checkpoint hits the validation definition store?

assert len(checkpoint.actions) == 3
assert isinstance(checkpoint.actions[0], UpdateDataDocsAction)
assert isinstance(checkpoint.actions[1], SlackNotificationAction)
assert isinstance(checkpoint.actions[2], MicrosoftTeamsNotificationAction)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd probably add another test here to verify that (pseudocode) serialized_checkpoint == serialize(deserialize(checkpoint))

Comment on lines 9 to 18
from great_expectations.checkpoint.actions import (
EmailAction, # noqa: TCH001
MicrosoftTeamsNotificationAction, # noqa: TCH001
OpsgenieAlertAction, # noqa: TCH001
PagerdutyAlertAction, # noqa: TCH001
SlackNotificationAction, # noqa: TCH001
SNSNotificationAction, # noqa: TCH001
StoreValidationResultAction, # noqa: TCH001
UpdateDataDocsAction, # noqa: TCH001
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than ignoring all these warnings I would add these (or their immediate parent) here.

runtime-evaluated-base-classes = [
# NOTE: ruff is unable to detect that these are subclasses of pydantic.BaseModel
"pydantic.BaseModel",
"great_expectations.datasource.fluent.fluent_base_model.FluentBaseModel",
"great_expectations.datasource.fluent.interfaces.Datasource",
"great_expectations.datasource.fluent.sql_datasource.SQLDatasource",
]

Copy link
Member

@Kilo59 Kilo59 Apr 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually sorry, you need Checkpoint here, or pydantic.v1.BaseModel not these actions.

"pydantic.BaseModel",
"pydantic.v1.BaseModel",
"great_expectations.compatibility.pydantic.BaseModel",
# if adding these doesn't fix it you can add the Checkpoint model directly.

Copy link

codecov bot commented Apr 3, 2024

Codecov Report

Attention: Patch coverage is 80.00000% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 82.56%. Comparing base (7993a68) to head (08247f3).

Files Patch % Lines
great_expectations/checkpoint/v1_checkpoint.py 75.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #9701   +/-   ##
========================================
  Coverage    82.55%   82.56%           
========================================
  Files          511      511           
  Lines        46454    46450    -4     
========================================
+ Hits         38351    38352    +1     
+ Misses        8103     8098    -5     
Flag Coverage Δ
3.10 64.61% <80.00%> (+0.01%) ⬆️
3.11 64.61% <80.00%> (+0.01%) ⬆️
3.11 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.95% <80.00%> (+<0.01%) ⬆️
3.11 aws_deps 48.99% <80.00%> (-0.01%) ⬇️
3.11 big 63.96% <80.00%> (+<0.01%) ⬆️
3.11 databricks 48.20% <80.00%> (-0.01%) ⬇️
3.11 filesystem 63.85% <80.00%> (+0.06%) ⬆️
3.11 mssql 47.42% <80.00%> (-0.01%) ⬇️
3.11 mysql 47.47% <80.00%> (-0.01%) ⬇️
3.11 postgresql 54.25% <80.00%> (+<0.01%) ⬆️
3.11 snowflake 48.73% <80.00%> (-0.01%) ⬇️
3.11 spark 60.64% <80.00%> (+<0.01%) ⬆️
3.11 trino 53.88% <80.00%> (+<0.01%) ⬆️
3.8 64.62% <80.00%> (+0.01%) ⬆️
3.8 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.96% <80.00%> (+<0.01%) ⬆️
3.8 aws_deps 49.00% <80.00%> (-0.01%) ⬇️
3.8 big 63.97% <80.00%> (+<0.01%) ⬆️
3.8 databricks 48.21% <80.00%> (-0.01%) ⬇️
3.8 filesystem 63.85% <80.00%> (+0.06%) ⬆️
3.8 mssql 47.40% <80.00%> (-0.01%) ⬇️
3.8 mysql 47.45% <80.00%> (-0.01%) ⬇️
3.8 postgresql 54.24% <80.00%> (+<0.01%) ⬆️
3.8 snowflake 48.74% <80.00%> (-0.01%) ⬇️
3.8 spark 60.60% <80.00%> (+<0.01%) ⬆️
3.8 trino 53.87% <80.00%> (+<0.01%) ⬆️
3.9 64.62% <80.00%> (+0.02%) ⬆️
cloud 0.00% <0.00%> (ø)
docs-basic 54.48% <80.00%> (+<0.01%) ⬆️
docs-creds-needed 55.05% <80.00%> (+<0.01%) ⬆️
docs-spark 54.57% <80.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@cdkini cdkini enabled auto-merge April 3, 2024 20:47
@cdkini cdkini added this pull request to the merge queue Apr 3, 2024
Merged via the queue into develop with commit 045d3e1 Apr 3, 2024
69 of 70 checks passed
@cdkini cdkini deleted the b/_/patch_checkpoint_serialization branch April 3, 2024 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants