Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MAINTENANCE] Migrate OpsgenieNotificationAction #9716

Merged
merged 18 commits into from Apr 8, 2024

Conversation

cdkini
Copy link
Member

@cdkini cdkini commented Apr 5, 2024

Opsgenie renderer and action need to support V1's checkpoint result API

  • Description of PR changes above includes a link to an existing GitHub issue
  • PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB]
  • Code is linted - run invoke lint (uses ruff format + ruff check)
  • Appropriate tests and docs have been updated

For more information about contributing, see Contribute.

After you submit your PR, keep the page open and monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up and reach out on Slack if you need help. Thanks for contributing!

Copy link

netlify bot commented Apr 5, 2024

Deploy Preview for niobium-lead-7998 canceled.

Name Link
🔨 Latest commit bf93776
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/66144161a7eca4000873de37

Copy link

codecov bot commented Apr 7, 2024

Codecov Report

Attention: Patch coverage is 89.04110% with 8 lines in your changes are missing coverage. Please review.

Project coverage is 82.64%. Comparing base (a75eaa1) to head (bf93776).

Files Patch % Lines
...expectations/checkpoint/types/checkpoint_result.py 0.00% 2 Missing ⚠️
great_expectations/checkpoint/util.py 77.77% 2 Missing ⚠️
..._expectations/render/renderer/opsgenie_renderer.py 93.10% 2 Missing ⚠️
great_expectations/checkpoint/actions.py 96.15% 1 Missing ⚠️
...expectations/core/expectation_validation_result.py 80.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #9716      +/-   ##
===========================================
+ Coverage    82.62%   82.64%   +0.01%     
===========================================
  Files          512      512              
  Lines        46532    46581      +49     
===========================================
+ Hits         38446    38495      +49     
  Misses        8086     8086              
Flag Coverage Δ
3.10 64.86% <72.60%> (+0.05%) ⬆️
3.10 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.10 aws_deps ?
3.10 big ?
3.10 databricks ?
3.10 filesystem ?
3.10 mssql ?
3.10 mysql ?
3.10 postgresql ?
3.10 spark ?
3.10 trino ?
3.11 64.86% <72.60%> (+0.05%) ⬆️
3.11 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.89% <16.43%> (-0.04%) ⬇️
3.11 aws_deps 48.97% <16.43%> (-0.03%) ⬇️
3.11 big 63.92% <39.72%> (-0.04%) ⬇️
3.11 databricks 48.15% <16.43%> (-0.03%) ⬇️
3.11 filesystem 63.80% <19.17%> (-0.05%) ⬇️
3.11 mssql 47.37% <16.43%> (-0.03%) ⬇️
3.11 mysql 47.42% <16.43%> (-0.03%) ⬇️
3.11 postgresql 54.18% <16.43%> (-0.04%) ⬇️
3.11 snowflake 48.68% <16.43%> (-0.03%) ⬇️
3.11 spark 60.59% <16.43%> (-0.05%) ⬇️
3.11 trino 53.81% <16.43%> (-0.04%) ⬇️
3.8 64.87% <72.60%> (+0.05%) ⬆️
3.8 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.89% <16.43%> (-0.04%) ⬇️
3.8 aws_deps 48.98% <16.43%> (-0.03%) ⬇️
3.8 big 63.93% <39.72%> (-0.04%) ⬇️
3.8 databricks 48.16% <16.43%> (-0.03%) ⬇️
3.8 filesystem 63.80% <19.17%> (-0.05%) ⬇️
3.8 mssql 47.35% <16.43%> (-0.03%) ⬇️
3.8 mysql 47.40% <16.43%> (-0.03%) ⬇️
3.8 postgresql 54.17% <16.43%> (-0.04%) ⬇️
3.8 snowflake 48.69% <16.43%> (-0.03%) ⬇️
3.8 spark 60.55% <16.43%> (-0.05%) ⬇️
3.8 trino 53.80% <16.43%> (-0.04%) ⬇️
3.9 64.86% <72.60%> (+0.05%) ⬆️
3.9 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.9 aws_deps ?
3.9 big ?
3.9 databricks ?
3.9 filesystem ?
3.9 mssql ?
3.9 mysql ?
3.9 postgresql ?
3.9 spark ?
3.9 trino ?
cloud 0.00% <0.00%> (ø)
docs-basic 54.45% <19.17%> (-0.04%) ⬇️
docs-creds-needed 55.02% <19.17%> (-0.04%) ⬇️
docs-spark 54.55% <19.17%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Comment on lines +291 to +294
assets_validated_by_batch_id[batch_id]["validation_results"].append( # type: ignore[union-attr]
validation_result
)
assets_validated_by_batch_id[batch_id]["expectation_suite_names"].append(
assets_validated_by_batch_id[batch_id]["expectation_suite_names"].append( # type: ignore[union-attr]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the old checkpoint result (not sure why mypy is mad about it now). We're using a new Pydantic type so I think we can ignore these?

@@ -514,7 +514,7 @@ def __init__( # noqa: PLR0913
suite_name: str,
evaluation_parameters: Optional[dict] = None,
statistics: Optional[dict] = None,
meta: Optional[ExpectationSuiteValidationResult | dict] = None,
meta: Optional[ExpectationSuiteValidationResultMeta | dict] = None,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has been wrong for a while (thank you @tyler-hoffman for pointing it out). Some minor changes in this PR to adhere to the proper type

Comment on lines +535 to +539
@property
def asset_name(self) -> str | None:
if "active_batch_definition" in self.meta:
return self.meta["active_batch_definition"].data_asset_name
return None
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would like to make this a top-level property (might write a ticket around it) but not pressing

@cdkini cdkini self-assigned this Apr 8, 2024
Copy link
Contributor

@tyler-hoffman tyler-hoffman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending that one private method rename

checkpoint_name = checkpoint_result.checkpoint_config.name

if (
self.notify_on == "all"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about this in the other PR but didn't mention it because it was just reusing existing code: maybe it's just me but I find these longer lines of or / and hard to parse in python. What about making intermediate variables for the and parts, and then just having this condition do the ors if that makes sense.

@@ -606,6 +606,41 @@ def _validate_renderer(cls, renderer: dict | OpsgenieRenderer) -> OpsgenieRender
renderer = _renderer
return renderer

@override
def v1_run(self, checkpoint_result: CheckpointResult) -> dict:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lots of overlap with _run. I might DRY this up, but if we can remove _run in the near term, I'm less stressed about it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Legacy _run will be removed shortly!

@@ -1044,19 +1080,8 @@ def _run( # type: ignore[override] # signature does not match parent
checkpoint_identifier=None,
**kwargs,
):
suite_name: str = validation_result_suite.meta["expectation_suite_name"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused on this change. This is a separate action, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup but mypy kept flagging it as an issue

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably because of the change to meta below

checkpoint_result=checkpoint_result, text_blocks=text_blocks
)

def _v1_render(self, run_result: ExpectationSuiteValidationResult) -> str:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be named to indicate it's just about a single result; first look at this method and the above made me question why one was public and one was private, and why they would be different.

@cdkini cdkini enabled auto-merge April 8, 2024 19:11
@cdkini cdkini added this pull request to the merge queue Apr 8, 2024
Merged via the queue into develop with commit 0bb4542 Apr 8, 2024
70 checks passed
@cdkini cdkini deleted the m/v1-251/opsgenie_action branch April 8, 2024 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants