Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document recommendations for retrying blob.upload_from_filename #352

Closed
tschaub opened this issue Dec 23, 2020 · 1 comment
Closed

Document recommendations for retrying blob.upload_from_filename #352

tschaub opened this issue Dec 23, 2020 · 1 comment

Comments

@tschaub
Copy link

@tschaub tschaub commented Dec 23, 2020

I'm seeing frequent 503 and 504 responses using the blob.upload_from_filename() method. An example trace looks like this:

503 POST https://storage.googleapis.com/upload/storage/v1/b/my-bucket/o?uploadType=multipart: {
  "error": {
    "code": 503,
    "message": "Backend Error",
    "errors": [
      {
        "message": "Backend Error",
        "domain": "global",
        "reason": "backendError"
      }
    ]
  }
}
: (u'Request failed with status code', 503, u'Expected one of', 200)' - Calling rollback() ...
Traceback (most recent call last):
  File "/my/app/module.py", line 112, in upload
    blob.upload_from_filename(file_path)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py", line 2471, in upload_from_filename
    checksum=checksum,
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py", line 2359, in upload_from_file
    _raise_from_invalid_response(exc)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py", line 3886, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
ServiceUnavailable: 503 POST https://storage.googleapis.com/upload/storage/v1/b/my-bucket/o?uploadType=multipart: {
  "error": {
    "code": 503,
    "message": "Backend Error",
    "errors": [
      {
        "message": "Backend Error",
        "domain": "global",
        "reason": "backendError"
      }
    ]
  }
}
: (u'Request failed with status code', 503, u'Expected one of', 200)

It looks like this method calls blob.upload_from_file() with num_retries=None, so there is no retry strategy applied. I also see that the num_retries argument is documented as deprecated.

Does this library provide an alternative way to retry blob.upload_from_filename() or blob.upload_from_file()? I can wrap these calls in retry logic of my own, but I see that retries are enabled by default for other functions.

I'm using google-cloud-storage==1.34.0. I've already pulled in google-resumable-media==1.20.0 so I should be getting the recent resumable media retry fixes that came with #342.

Thanks for any tips on retrying uploads.

@HemangChothani
Copy link
Contributor

@HemangChothani HemangChothani commented Dec 23, 2020

@tschaub See the documentation of num_retries here

By default, only uploads with if_metageneration_match set will be retried for the transient errors.For now num_retires parameter working fine and it will remove in the future so you can use that.
The num_retries is not exposed for blob.upload_from_filename method so for now you can use blob.upload_from_file method.

google-resumable-media==1.2.0 automatically pulled when you update your storage library to google-cloud-storage==1.35.0 and you will get the recent resumable media retry fix with it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

2 participants