Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Update expectations and checkpoints v1 stores to implement gx_cloud_response_json_to_object_collection #9718

Merged
merged 6 commits into from Apr 8, 2024

Conversation

tyler-hoffman
Copy link
Contributor

@tyler-hoffman tyler-hoffman commented Apr 5, 2024

Goals

  • Unlock the ability for our existing factories to implement all, which depends on their stores' get_all, which in turn relies on the stores' gx_cloud_response_json_to_object_collection
  • Somewhat formalize a pattern that we can adopt more widely about how to do this

Implementation

  • Move gx_cloud_response_json_to_object_collection into the parent store
  • Put a not implemented _convert_raw_json_to_object_dict in the parent store
  • Override _convert_raw_json_to_object_dict in the 2 new stores we care about

What this does not do

  • We have a factory for ValidationDefinition, but we cannot yet implement gx_cloud_response_json_to_object_collection for that store because we don't yet have a REST endpoint
  • This also leaves most stores in a state where they don't have the functionality implemented. They will get a NotImplemented error when calling get_all, but that behavior is the same as before (except now it raises from one step deeper in the call stack, in _convert_raw_json_to_object_dict
  • Totally fix our inconsistencies. We should eventually move this logic into the cloud store backend, but we aren't there today.
  • Description of PR changes above includes a link to an existing GitHub issue
  • PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB]
  • Code is linted - run invoke lint (uses ruff format + ruff check)
  • Appropriate tests and docs have been updated

For more information about contributing, see Contribute.

After you submit your PR, keep the page open and monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up and reach out on Slack if you need help. Thanks for contributing!

Copy link

netlify bot commented Apr 5, 2024

Deploy Preview for niobium-lead-7998 canceled.

Name Link
🔨 Latest commit f70ce54
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/66133a7e51718d0008839438

@tyler-hoffman tyler-hoffman requested review from joshua-stauffer, cdkini and billdirks and removed request for joshua-stauffer April 5, 2024 20:51
Copy link

codecov bot commented Apr 5, 2024

Codecov Report

Attention: Patch coverage is 96.77419% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 82.57%. Comparing base (b3d4769) to head (f70ce54).

Files Patch % Lines
great_expectations/data_context/store/store.py 91.66% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #9718      +/-   ##
===========================================
+ Coverage    82.55%   82.57%   +0.02%     
===========================================
  Files          511      511              
  Lines        46482    46491       +9     
===========================================
+ Hits         38372    38392      +20     
+ Misses        8110     8099      -11     
Flag Coverage Δ
3.10 64.67% <96.77%> (+0.04%) ⬆️
3.10 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.10 aws_deps ?
3.10 big ?
3.10 databricks ?
3.10 filesystem ?
3.10 mssql ?
3.10 mysql ?
3.10 postgresql ?
3.10 snowflake ?
3.10 spark ?
3.10 trino ?
3.11 64.67% <96.77%> (+0.04%) ⬆️
3.11 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.93% <64.51%> (+<0.01%) ⬆️
3.11 aws_deps 49.00% <64.51%> (+<0.01%) ⬆️
3.11 big 63.96% <64.51%> (+<0.01%) ⬆️
3.11 databricks 48.18% <64.51%> (+<0.01%) ⬆️
3.11 filesystem 63.86% <74.19%> (+<0.01%) ⬆️
3.11 mssql 47.40% <64.51%> (+<0.01%) ⬆️
3.11 mysql 47.46% <64.51%> (+<0.01%) ⬆️
3.11 postgresql 54.22% <64.51%> (+<0.01%) ⬆️
3.11 snowflake 48.71% <64.51%> (+<0.01%) ⬆️
3.11 spark 60.65% <77.41%> (+<0.01%) ⬆️
3.11 trino 53.86% <64.51%> (+<0.01%) ⬆️
3.8 64.68% <96.77%> (+0.04%) ⬆️
3.8 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.94% <64.51%> (+<0.01%) ⬆️
3.8 aws_deps 49.02% <64.51%> (+<0.01%) ⬆️
3.8 big 63.96% <64.51%> (+<0.01%) ⬆️
3.8 databricks 48.20% <64.51%> (+<0.01%) ⬆️
3.8 filesystem 63.87% <74.19%> (+<0.01%) ⬆️
3.8 mssql 47.39% <64.51%> (+<0.01%) ⬆️
3.8 mysql 47.44% <64.51%> (+<0.01%) ⬆️
3.8 postgresql 54.21% <64.51%> (+<0.01%) ⬆️
3.8 snowflake 48.73% <64.51%> (+<0.01%) ⬆️
3.8 spark 60.61% <77.41%> (+<0.01%) ⬆️
3.8 trino 53.85% <64.51%> (+<0.01%) ⬆️
3.9 64.68% <96.77%> (+0.04%) ⬆️
3.9 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.9 aws_deps ?
3.9 big ?
3.9 databricks ?
3.9 filesystem ?
3.9 mssql ?
3.9 mysql ?
3.9 postgresql ?
3.9 snowflake ?
3.9 spark ?
3.9 trino ?
cloud 0.00% <0.00%> (ø)
docs-basic 54.50% <64.51%> (+<0.01%) ⬆️
docs-creds-needed 55.07% <64.51%> (+<0.01%) ⬆️
docs-spark 54.60% <64.51%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tyler-hoffman tyler-hoffman mentioned this pull request Apr 8, 2024
4 tasks
@tyler-hoffman tyler-hoffman added this pull request to the merge queue Apr 8, 2024
Copy link
Member

@joshua-stauffer joshua-stauffer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, thanks!

Merged via the queue into develop with commit 71ffbe1 Apr 8, 2024
69 of 70 checks passed
@tyler-hoffman tyler-hoffman deleted the f/v1-231/store-collection-methods branch April 8, 2024 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants