Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Checkpoint.save() #9676

Merged
merged 16 commits into from Apr 1, 2024
Merged

[FEATURE] Checkpoint.save() #9676

merged 16 commits into from Apr 1, 2024

Conversation

cdkini
Copy link
Member

@cdkini cdkini commented Mar 28, 2024

  • Description of PR changes above includes a link to an existing GitHub issue
  • PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB]
  • Code is linted - run invoke lint (uses ruff format + ruff check)
  • Appropriate tests and docs have been updated

For more information about contributing, see Contribute.

After you submit your PR, keep the page open and monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up and reach out on Slack if you need help. Thanks for contributing!

@cdkini cdkini self-assigned this Mar 28, 2024
Copy link

netlify bot commented Mar 28, 2024

Deploy Preview for niobium-lead-7998 canceled.

Name Link
🔨 Latest commit 35457a7
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/660af241f6e3e50008ba3c96

Copy link

codecov bot commented Mar 28, 2024

Codecov Report

Attention: Patch coverage is 95.45455% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 82.55%. Comparing base (ff5cc98) to head (35457a7).

Files Patch % Lines
...tions/data_context/data_context/context_factory.py 66.66% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #9676   +/-   ##
========================================
  Coverage    82.55%   82.55%           
========================================
  Files          511      511           
  Lines        46381    46395   +14     
========================================
+ Hits         38288    38301   +13     
- Misses        8093     8094    +1     
Flag Coverage Δ
3.10 64.61% <95.45%> (+0.01%) ⬆️
3.11 64.61% <95.45%> (+0.01%) ⬆️
3.11 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.97% <50.00%> (-0.01%) ⬇️
3.11 aws_deps 48.98% <50.00%> (-0.01%) ⬇️
3.11 big 63.99% <50.00%> (-0.01%) ⬇️
3.11 databricks 48.21% <50.00%> (-0.01%) ⬇️
3.11 filesystem 63.78% <50.00%> (-0.01%) ⬇️
3.11 mssql 47.43% <50.00%> (-0.01%) ⬇️
3.11 mysql 47.48% <50.00%> (-0.01%) ⬇️
3.11 postgresql 54.26% <50.00%> (-0.01%) ⬇️
3.11 snowflake 48.74% <50.00%> (-0.01%) ⬇️
3.11 spark 60.65% <50.00%> (-0.01%) ⬇️
3.11 trino 53.90% <50.00%> (-0.01%) ⬇️
3.8 64.61% <95.45%> (-0.01%) ⬇️
3.8 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.98% <50.00%> (-0.01%) ⬇️
3.8 aws_deps 49.00% <50.00%> (-0.01%) ⬇️
3.8 big 63.99% <50.00%> (-0.01%) ⬇️
3.8 databricks 48.22% <50.00%> (-0.01%) ⬇️
3.8 filesystem 63.78% <50.00%> (-0.01%) ⬇️
3.8 mssql 47.41% <50.00%> (-0.01%) ⬇️
3.8 mysql 47.46% <50.00%> (-0.01%) ⬇️
3.8 postgresql 54.25% <50.00%> (-0.01%) ⬇️
3.8 snowflake 48.76% <50.00%> (-0.01%) ⬇️
3.8 spark 60.61% <50.00%> (-0.01%) ⬇️
3.8 trino 53.89% <50.00%> (-0.01%) ⬇️
3.9 64.62% <95.45%> (+0.01%) ⬆️
cloud 0.00% <0.00%> (ø)
docs-basic 54.49% <50.00%> (-0.01%) ⬇️
docs-creds-needed 55.07% <50.00%> (-0.01%) ⬇️
docs-spark 54.59% <50.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

try:
store.update(key=key, value=self)
except gx_exceptions.StoreBackendError as e:
raise ValueError(f"Could not find existing Checkpoint '{self.name}' to update") from e
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this error is too specific - for instance, if the cloud handler returns a 400 due to a field of an unexpected type.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm what error do we want to catch then? The other domain objects don't do error handling here but I don't want to raise a store backend error to the user (feels like that is more internal?)

@@ -21,9 +18,8 @@

# TODO: Add analytics as needed
class CheckpointFactory(Factory[Checkpoint]):
def __init__(self, store: CheckpointStore, context: AbstractDataContext):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥳

Copy link
Member

@joshua-stauffer joshua-stauffer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left one comment around error handling

Comment on lines 301 to 303
except gx_exceptions.StoreBackendError as e:
name = key.to_tuple()[0]
raise ValueError(f"Could not find existing Checkpoint '{name}' to update") from e
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't this the same issue as before? There are other valid errors -- such as validation -- Cloud backend can throw, so this error message is too specific.

Copy link
Member

@joshua-stauffer joshua-stauffer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocking because i believe error handling is still misleading

Copy link
Member

@joshua-stauffer joshua-stauffer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for addressing changes!

@cdkini cdkini added this pull request to the merge queue Apr 1, 2024
Merged via the queue into develop with commit ee66aac Apr 1, 2024
191 of 217 checks passed
@cdkini cdkini deleted the m/v1-268/checkpoint_save branch April 1, 2024 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants