Skip to content
Permalink
Browse files
docs: streamline 'timeout' / 'retry' docs in docstrings (#461)
* Add 'requests' intersphinx refs
* Add narrative docs for timeouts and retries
* Include API docs for the 'retry' module, as well, to unbreak links.
* Replace boilerplate docstring entries for 'timeout' / 'retry' with links to new narrative docs.
* Add docstrings for default conditional policies.

Closes #455.
  • Loading branch information
tseaver committed Jun 10, 2021
1 parent 2c87f2f commit 78b2eba81003b437cd24f2b8d269ea2455682507
@@ -364,6 +364,7 @@
"grpc": ("https://grpc.github.io/grpc/python/", None),
"proto-plus": ("https://proto-plus-python.readthedocs.io/en/latest/", None),
"protobuf": ("https://googleapis.dev/python/protobuf/latest/", None),
"requests": ("https://docs.python-requests.org/en/master/", None),
}


@@ -21,6 +21,7 @@ API Reference
constants
hmac_key
notification
retry_timeout

Changelog
---------
@@ -0,0 +1,152 @@
Configuring Timeouts and Retries
================================

When using object methods which invoke Google Cloud Storage API methods,
you have several options for how the library handles timeouts and
how it retries transient errors.


.. _configuring_timeouts:

Configuring Timeouts
--------------------

For a number of reasons, methods which invoke API methods may take
longer than expected or desired. By default, such methods all time out
after a default interval, 60.0 seconds. Rather than blocking your application
code for that interval, you may choose to configure explicit timeouts
in your code, using one of three forms:

- You can pass a single integer or float which functions as the timeout for the
entire request. E.g.:

.. code-block:: python
bucket = client.get_bucket(BUCKET_NAME, timeout=300.0) # five minutes
- You can also be passed as a two-tuple, ``(connect_timeout, read_timeout)``,
where the ``connect_timeout`` sets the maximum time required to establish
the connection to the server, and the ``read_timeout`` sets the maximum
time to wait for a completed response. E.g.:

.. code-block:: python
bucket = client.get_bucket(BUCKET_NAME, timeout=(3, 10))
- You can also pass ``None`` as the timeout value: in this case, the library
will block indefinitely for a response. E.g.:

.. code-block:: python
bucket = client.get_bucket(BUCKET_NAME, timeout=None)
.. note::
Depending on the retry strategy, a request may be
repeated several times using the same timeout each time.

See also:

:ref:`Timeouts in requests <requests:timeouts>`


.. _configuring_retries:

Configuring Retries
--------------------

.. note::

For more background on retries, see also the
`GCS Retry Strategies Document <https://cloud.google.com/storage/docs/retry-strategy#python>`_

Methods which invoke API methods may fail for a number of reasons, some of
which represent "transient" conditions, and thus can be retried
automatically. The library tries to provide a sensible default retry policy
for each method, base on its semantics:

- For API requests which are always idempotent, the library uses its
:data:`~google.cloud.storage.retry.DEFAULT_RETRY` policy, which
retries any API request which returns a "transient" error.

- For API requests which are idempotent only if the blob has
the same "generation", the library uses its
:data:`~google.cloud.storage.retry.DEFAULT_RETRY_IF_GENERATION_SPECIFIED`
policy, which retries API requests which returns a "transient" error,
but only if the original request includes an ``ifGenerationMatch`` header.

- For API requests which are idempotent only if the bucket or blob has
the same "metageneration", the library uses its
:data:`~google.cloud.storage.retry.DEFAULT_RETRY_IF_METAGENERATION_SPECIFIED`
policy, which retries API requests which returns a "transient" error,
but only if the original request includes an ``ifMetagenerationMatch`` header.

- For API requests which are idempotent only if the bucket or blob has
the same "etag", the library uses its
:data:`~google.cloud.storage.retry.DEFAULT_RETRY_IF_ETAG_IN_JSON`
policy, which retries API requests which returns a "transient" error,
but only if the original request includes an ``ETAG`` in its payload.

- For those API requests which are never idempotent, the library passes
``retry=None`` by default, suppressing any retries.

Rather than using one of the default policies, you may choose to configure an
explicit policy in your code.

- You can pass ``None`` as a retry policy to disable retries. E.g.:

.. code-block:: python
bucket = client.get_bucket(BUCKET_NAME, retry=None)
- You can pass an instance of :class:`google.api_core.retry.Retry` to enable
retries; the passed object will define retriable response codes and errors,
as well as configuring backoff and retry interval options. E.g.:

.. code-block:: python
from google.api_core import exceptions
from google.api_core.retry import Retry
_MY_RETRIABLE_TYPES = [
exceptions.TooManyRequests, # 429
exceptions.InternalServerError, # 500
exceptions.BadGateway, # 502
exceptions.ServiceUnavailable, # 503
]
def is_retryable(exc):
return isinstance(exc, _MY_RETRIABLE_TYPES)
my_retry_policy = Retry(predicate=is_retryable)
bucket = client.get_bucket(BUCKET_NAME, retry=my_retry_policy)
- You can pass an instance of
:class:`google.cloud.storage.retry.ConditionalRetryPolicy`, which wraps a
:class:`~google.cloud.storage.retry.RetryPolicy`, activating it only if
certain conditions are met. This class exists to provide safe defaults
for RPC calls that are not technically safe to retry normally (due to
potential data duplication or other side-effects) but become safe to retry
if a condition such as if_metageneration_match is set. E.g.:

.. code-block:: python
from google.api_core.retry import Retry
from google.cloud.storage.retry import ConditionalRetryPolicy
from google.cloud.storage.retry import is_etag_in_json
def is_retryable(exc):
... # as above
my_retry_policy = Retry(predicate=is_retryable)
my_cond_policy = ConditionalRetryPolicy(
my_retry_policy, conditional_predicate=is_etag_in_json)
bucket = client.get_bucket(BUCKET_NAME, retry=my_cond_policy)
Retry Module API
----------------

.. automodule:: google.cloud.storage.retry
:members:
:show-inheritance:
@@ -162,11 +162,9 @@ def reload(
properties to return.
:type timeout: float or tuple
:param timeout: (Optional) The amount of time, in seconds, to wait
for the server response.
Can also be passed as a tuple (connect_timeout, read_timeout).
See :meth:`requests.Session.request` documentation for details.
:param timeout:
(Optional) The amount of time, in seconds, to wait
for the server response. See: :ref:`configuring_timeouts`
:type if_generation_match: long
:param if_generation_match: (Optional) Make the operation conditional on whether
@@ -190,18 +188,8 @@ def reload(
blob's current metageneration does not match the given value.
:type retry: google.api_core.retry.Retry or google.cloud.storage.retry.ConditionalRetryPolicy
:param retry: (Optional) How to retry the RPC. A None value will disable retries.
A google.api_core.retry.Retry value will enable retries, and the object will
define retriable response codes and errors and configure backoff and timeout options.
A google.cloud.storage.retry.ConditionalRetryPolicy value wraps a Retry object and
activates it only if certain conditions are met. This class exists to provide safe defaults
for RPC calls that are not technically safe to retry normally (due to potential data
duplication or other side-effects) but become safe to retry if a condition such as
if_metageneration_match is set.
See the retry.py source code and docstrings in this package (google.cloud.storage.retry) for
information on retry types and how to configure them.
:param retry:
(Optional) How to retry the RPC. See: :ref:`configuring_retries`
"""
client = self._require_client(client)
query_params = self._query_params
@@ -275,11 +263,9 @@ def patch(
``client`` stored on the current object.
:type timeout: float or tuple
:param timeout: (Optional) The amount of time, in seconds, to wait
for the server response.
Can also be passed as a tuple (connect_timeout, read_timeout).
See :meth:`requests.Session.request` documentation for details.
:param timeout:
(Optional) The amount of time, in seconds, to wait
for the server response. See: :ref:`configuring_timeouts`
:type if_generation_match: long
:param if_generation_match: (Optional) Make the operation conditional on whether
@@ -303,18 +289,8 @@ def patch(
blob's current metageneration does not match the given value.
:type retry: google.api_core.retry.Retry or google.cloud.storage.retry.ConditionalRetryPolicy
:param retry: (Optional) How to retry the RPC. A None value will disable retries.
A google.api_core.retry.Retry value will enable retries, and the object will
define retriable response codes and errors and configure backoff and timeout options.
A google.cloud.storage.retry.ConditionalRetryPolicy value wraps a Retry object and
activates it only if certain conditions are met. This class exists to provide safe defaults
for RPC calls that are not technically safe to retry normally (due to potential data
duplication or other side-effects) but become safe to retry if a condition such as
if_metageneration_match is set.
See the retry.py source code and docstrings in this package (google.cloud.storage.retry) for
information on retry types and how to configure them.
:param retry:
(Optional) How to retry the RPC. See: :ref:`configuring_retries`
"""
client = self._require_client(client)
query_params = self._query_params
@@ -363,11 +339,9 @@ def update(
``client`` stored on the current object.
:type timeout: float or tuple
:param timeout: (Optional) The amount of time, in seconds, to wait
for the server response.
Can also be passed as a tuple (connect_timeout, read_timeout).
See :meth:`requests.Session.request` documentation for details.
:param timeout:
(Optional) The amount of time, in seconds, to wait
for the server response. See: :ref:`configuring_timeouts`
:type if_generation_match: long
:param if_generation_match: (Optional) Make the operation conditional on whether
@@ -391,18 +365,8 @@ def update(
blob's current metageneration does not match the given value.
:type retry: google.api_core.retry.Retry or google.cloud.storage.retry.ConditionalRetryPolicy
:param retry: (Optional) How to retry the RPC. A None value will disable retries.
A google.api_core.retry.Retry value will enable retries, and the object will
define retriable response codes and errors and configure backoff and timeout options.
A google.cloud.storage.retry.ConditionalRetryPolicy value wraps a Retry object and
activates it only if certain conditions are met. This class exists to provide safe defaults
for RPC calls that are not technically safe to retry normally (due to potential data
duplication or other side-effects) but become safe to retry if a condition such as
if_metageneration_match is set.
See the retry.py source code and docstrings in this package (google.cloud.storage.retry) for
information on retry types and how to configure them.
:param retry:
(Optional) How to retry the RPC. See: :ref:`configuring_retries`
"""
client = self._require_client(client)

@@ -219,11 +219,9 @@ def _ensure_loaded(self, timeout=_DEFAULT_TIMEOUT):
"""Load if not already loaded.
:type timeout: float or tuple
:param timeout: (Optional) The amount of time, in seconds, to wait
for the server response.
Can also be passed as a tuple (connect_timeout, read_timeout).
See :meth:`requests.Session.request` documentation for details.
:param timeout:
(Optional) The amount of time, in seconds, to wait
for the server response. See: :ref:`configuring_timeouts`
"""
if not self.loaded:
self.reload(timeout=timeout)
@@ -442,20 +440,13 @@ def reload(self, client=None, timeout=_DEFAULT_TIMEOUT, retry=DEFAULT_RETRY):
:param client: (Optional) The client to use. If not passed, falls back
to the ``client`` stored on the ACL's parent.
:type timeout: float or tuple
:param timeout: (Optional) The amount of time, in seconds, to wait
for the server response.
Can also be passed as a tuple (connect_timeout, read_timeout).
See :meth:`requests.Session.request` documentation for details.
:param timeout:
(Optional) The amount of time, in seconds, to wait
for the server response. See: :ref:`configuring_timeouts`
:type retry: :class:`~google.api_core.retry.Retry`
:param retry: (Optional) How to retry the RPC.
A None value will disable retries.
A google.api_core.retry.Retry value will enable retries,
and the object will define retriable response codes and errors
and configure backoff and timeout options.
:param retry:
(Optional) How to retry the RPC. See: :ref:`configuring_retries`
"""
path = self.reload_path
client = self._require_client(client)
@@ -489,21 +480,15 @@ def _save(self, acl, predefined, client, timeout=_DEFAULT_TIMEOUT):
``NoneType``
:param client: (Optional) The client to use. If not passed, falls back
to the ``client`` stored on the ACL's parent.
:type timeout: float or tuple
:param timeout: (Optional) The amount of time, in seconds, to wait
for the server response.
Can also be passed as a tuple (connect_timeout, read_timeout).
See :meth:`requests.Session.request` documentation for details.
:type timeout: float or tuple
:param timeout:
(Optional) The amount of time, in seconds, to wait
for the server response. See: :ref:`configuring_timeouts`
:type retry: :class:`~google.api_core.retry.Retry`
:param retry: (Optional) How to retry the RPC.
A None value will disable retries.
A google.api_core.retry.Retry value will enable retries,
and the object will define retriable response codes and errors
and configure backoff and timeout options.
:param retry:
(Optional) How to retry the RPC. See: :ref:`configuring_retries`
"""
client = self._require_client(client)
query_params = {"projection": "full"}
@@ -545,12 +530,11 @@ def save(self, acl=None, client=None, timeout=_DEFAULT_TIMEOUT):
``NoneType``
:param client: (Optional) The client to use. If not passed, falls back
to the ``client`` stored on the ACL's parent.
:type timeout: float or tuple
:param timeout: (Optional) The amount of time, in seconds, to wait
for the server response.
Can also be passed as a tuple (connect_timeout, read_timeout).
See :meth:`requests.Session.request` documentation for details.
:type timeout: float or tuple
:param timeout:
(Optional) The amount of time, in seconds, to wait
for the server response. See: :ref:`configuring_timeouts`
"""
if acl is None:
acl = self
@@ -577,12 +561,11 @@ def save_predefined(self, predefined, client=None, timeout=_DEFAULT_TIMEOUT):
``NoneType``
:param client: (Optional) The client to use. If not passed, falls back
to the ``client`` stored on the ACL's parent.
:type timeout: float or tuple
:param timeout: (Optional) The amount of time, in seconds, to wait
for the server response.
Can also be passed as a tuple (connect_timeout, read_timeout).
See :meth:`requests.Session.request` documentation for details.
:type timeout: float or tuple
:param timeout:
(Optional) The amount of time, in seconds, to wait
for the server response. See: :ref:`configuring_timeouts`
"""
predefined = self.validate_predefined(predefined)
self._save(None, predefined, client, timeout=timeout)
@@ -601,12 +584,11 @@ def clear(self, client=None, timeout=_DEFAULT_TIMEOUT):
``NoneType``
:param client: (Optional) The client to use. If not passed, falls back
to the ``client`` stored on the ACL's parent.
:type timeout: float or tuple
:param timeout: (Optional) The amount of time, in seconds, to wait
for the server response.
Can also be passed as a tuple (connect_timeout, read_timeout).
See :meth:`requests.Session.request` documentation for details.
:type timeout: float or tuple
:param timeout:
(Optional) The amount of time, in seconds, to wait
for the server response. See: :ref:`configuring_timeouts`
"""
self.save([], client=client, timeout=timeout)

0 comments on commit 78b2eba

Please sign in to comment.