Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add http.client.RemoteDisconnected to list of retryable exceptions #1231

Closed
andrewpollock opened this issue Feb 26, 2024 · 1 comment · Fixed by #1234
Closed

Add http.client.RemoteDisconnected to list of retryable exceptions #1231

andrewpollock opened this issue Feb 26, 2024 · 1 comment · Fixed by #1234
Assignees
Labels
api: storage Issues related to the googleapis/python-storage API. priority: p3 Desirable enhancement or fix. May not be included in next release. type: question Request for information or clarification. Not an issue.

Comments

@andrewpollock
Copy link

Environment details

  • OS type and version:
  • Python version: Python 3.11.4
  • pip version: pip 23.1.2 from /usr/local/lib/python3.11/site-packages/pip (python 3.11)
  • google-cloud-storage version: 2.14.0

Steps to reproduce

  1. Usage of Blob.upload_from_filename() seems to sometimes raise a RemoteDisconnected exception
  2. https://docs.python.org/3/library/http.client.html#http.client.RemoteDisconnected get caught by our exception handler (see below)
  3. It doesn't look like
    _RETRYABLE_TYPES = (
    api_exceptions.TooManyRequests, # 429
    api_exceptions.InternalServerError, # 500
    api_exceptions.BadGateway, # 502
    api_exceptions.ServiceUnavailable, # 503
    api_exceptions.GatewayTimeout, # 504
    ConnectionError,
    requests.ConnectionError,
    requests_exceptions.ChunkedEncodingError,
    requests_exceptions.Timeout,
    )
    will treat this as retryable?

I think this is in the same vein as #414 and #426

Code example

https://github.com/google/osv.dev/blob/b705d0d0b7450ce94137624118a2b54a7f719147/docker/exporter/exporter.py#L57-L64

  def upload_single(self, bucket, source_path, target_path):
    """Upload a single file to a bucket."""
    logging.info('Uploading %s', target_path)
    try:
      blob = bucket.blob(target_path)
      blob.upload_from_filename(source_path)
    except Exception as e:
      logging.error('Failed to export: %s', e)

Stack trace

Failed to export: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
@product-auto-label product-auto-label bot added the api: storage Issues related to the googleapis/python-storage API. label Feb 26, 2024
@cojenco cojenco self-assigned this Feb 26, 2024
@cojenco cojenco added type: question Request for information or clarification. Not an issue. priority: p3 Desirable enhancement or fix. May not be included in next release. labels Feb 28, 2024
@cojenco
Copy link
Contributor

cojenco commented Mar 15, 2024

Discuffed offline and tracked in #1232

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/python-storage API. priority: p3 Desirable enhancement or fix. May not be included in next release. type: question Request for information or clarification. Not an issue.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants