Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document recommendations for retrying blob.upload_from_filename #352

Closed
tschaub opened this issue Dec 23, 2020 · 1 comment · Fixed by #353
Closed

Document recommendations for retrying blob.upload_from_filename #352

tschaub opened this issue Dec 23, 2020 · 1 comment · Fixed by #353
Assignees
Labels
api: storage Issues related to the googleapis/python-storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@tschaub
Copy link

tschaub commented Dec 23, 2020

I'm seeing frequent 503 and 504 responses using the blob.upload_from_filename() method. An example trace looks like this:

503 POST https://storage.googleapis.com/upload/storage/v1/b/my-bucket/o?uploadType=multipart: {
  "error": {
    "code": 503,
    "message": "Backend Error",
    "errors": [
      {
        "message": "Backend Error",
        "domain": "global",
        "reason": "backendError"
      }
    ]
  }
}
: (u'Request failed with status code', 503, u'Expected one of', 200)' - Calling rollback() ...
Traceback (most recent call last):
  File "/my/app/module.py", line 112, in upload
    blob.upload_from_filename(file_path)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py", line 2471, in upload_from_filename
    checksum=checksum,
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py", line 2359, in upload_from_file
    _raise_from_invalid_response(exc)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py", line 3886, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
ServiceUnavailable: 503 POST https://storage.googleapis.com/upload/storage/v1/b/my-bucket/o?uploadType=multipart: {
  "error": {
    "code": 503,
    "message": "Backend Error",
    "errors": [
      {
        "message": "Backend Error",
        "domain": "global",
        "reason": "backendError"
      }
    ]
  }
}
: (u'Request failed with status code', 503, u'Expected one of', 200)

It looks like this method calls blob.upload_from_file() with num_retries=None, so there is no retry strategy applied. I also see that the num_retries argument is documented as deprecated.

Does this library provide an alternative way to retry blob.upload_from_filename() or blob.upload_from_file()? I can wrap these calls in retry logic of my own, but I see that retries are enabled by default for other functions.

I'm using google-cloud-storage==1.34.0. I've already pulled in google-resumable-media==1.20.0 so I should be getting the recent resumable media retry fixes that came with #342.

Thanks for any tips on retrying uploads.

@product-auto-label product-auto-label bot added the api: storage Issues related to the googleapis/python-storage API. label Dec 23, 2020
@HemangChothani
Copy link
Contributor

@tschaub See the documentation of num_retries here

By default, only uploads with if_metageneration_match set will be retried for the transient errors.For now num_retires parameter working fine and it will remove in the future so you can use that.
The num_retries is not exposed for blob.upload_from_filename method so for now you can use blob.upload_from_file method.

google-resumable-media==1.2.0 automatically pulled when you update your storage library to google-cloud-storage==1.35.0 and you will get the recent resumable media retry fix with it.

@HemangChothani HemangChothani added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Dec 23, 2020
@HemangChothani HemangChothani self-assigned this Dec 23, 2020
cojenco pushed a commit to cojenco/python-storage that referenced this issue Oct 13, 2021
cojenco pushed a commit to cojenco/python-storage that referenced this issue Oct 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/python-storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
2 participants