Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: replace default retry for upload operations #480

Merged
merged 11 commits into from Jun 30, 2021
Merged

Conversation

@cojenco
Copy link
Contributor

@cojenco cojenco commented Jun 25, 2021

This PR replaces upload operations default retries with storage.retry.DEFAULT_RETRY_IF_GENERATION_SPECIFIED and aligns with the retry strategy conditional idempotency.

With default retries set to storage.retry.DEFAULT_RETRY_IF_GENERATION_SPECIFIED, passing in ifGenerationMatch makes the upload operation (1) conditional and (2) only retried by default on whether the object's current generation matches the given value. Setting if_generation_match=0 makes the upload operation succeed only if there are no live versions of the object (fresh uploads) and supports retries.

  • revises upload operations both in entry methods and private methods
  • updates tests
  • revises docstrings
  • (update: separating this to #484 ) adds preconditions and retry configuration to an upload entry method blob.create_resumable_upload_session()

Fixes #477 馃

Copy link
Contributor

@andrewsg andrewsg left a comment

Good work here. Please amend the PR title to include that create_resumable_upload_session is being amended in the same PR (or split it off into a separate PR) so that the change is included in the release notes.

@@ -1750,7 +1749,7 @@ def _do_multipart_upload(
This private method does not accept ConditionalRetryPolicy values
because the information necessary to evaluate the policy is instead
evaluated in client.download_blob_to_file().
evaluated in blob._do_upload().
Copy link
Contributor

@andrewsg andrewsg Jun 29, 2021

Oh, it looks like we're evaluating it BOTH in blob._do_upload() and in client.download_blob_to_file(). This is harmless, which is why the tests didn't catch it, but we only need one of those.

I don't have a strong opinion on whether we do this in _do_upload or download_blob_to_file. Let's keep this docstring change and additionally we can remove the conditional retry policy code in client.download_blob_to_file, either in another PR, or in this one if you prefer.

Copy link
Contributor Author

@cojenco cojenco Jun 29, 2021

IIUC, I believe we'll need to remain both ConditionalRetryPolicy evaluations as one is for upload operations and the other is for downloads. I revised the docstrings here to match particularly where we're doing the evaluations for upload operations, blob._do_upload() .

Copy link
Contributor

@tritone tritone left a comment

Nice catch on this! A couple questions

google/cloud/storage/blob.py Show resolved Hide resolved
google/cloud/storage/blob.py Outdated Show resolved Hide resolved
tests/unit/test_blob.py Outdated Show resolved Hide resolved
@cojenco cojenco mentioned this pull request Jun 30, 2021
4 tasks
@cojenco cojenco merged commit c027ccf into master Jun 30, 2021
6 checks passed
@cojenco cojenco deleted the update-upload-retry-477 branch Jun 30, 2021
cojenco added a commit to cojenco/python-storage that referenced this issue Oct 13, 2021
* fix: revise upload operations preconditions to if_generation_match

* fix docstrings

* add retry configuration to blob.create_resumable_upload_session

* align var values in test

* test coverage

* Revert "test coverage"

This reverts commit e91916f.

* Revert "align var values in test"

This reverts commit aec585b.

* Revert "add retry configuration to blob.create_resumable_upload_session"

This reverts commit 8c1ae3c.

* revise tests after reverting
cojenco added a commit to cojenco/python-storage that referenced this issue Oct 13, 2021
* fix: revise upload operations preconditions to if_generation_match

* fix docstrings

* add retry configuration to blob.create_resumable_upload_session

* align var values in test

* test coverage

* Revert "test coverage"

This reverts commit e91916f.

* Revert "align var values in test"

This reverts commit aec585b.

* Revert "add retry configuration to blob.create_resumable_upload_session"

This reverts commit 8c1ae3c.

* revise tests after reverting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

3 participants