forked from googleapis/python-storage
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: streamline 'timeout' / 'retry' docs in docstrings (googleapis#461)
* Add 'requests' intersphinx refs * Add narrative docs for timeouts and retries * Include API docs for the 'retry' module, as well, to unbreak links. * Replace boilerplate docstring entries for 'timeout' / 'retry' with links to new narrative docs. * Add docstrings for default conditional policies. Closes googleapis#455.
- Loading branch information
Showing
13 changed files
with
450 additions
and
851 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -21,6 +21,7 @@ API Reference | |
constants | ||
hmac_key | ||
notification | ||
retry_timeout | ||
|
||
Changelog | ||
--------- | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,152 @@ | ||
Configuring Timeouts and Retries | ||
================================ | ||
|
||
When using object methods which invoke Google Cloud Storage API methods, | ||
you have several options for how the library handles timeouts and | ||
how it retries transient errors. | ||
|
||
|
||
.. _configuring_timeouts: | ||
|
||
Configuring Timeouts | ||
-------------------- | ||
|
||
For a number of reasons, methods which invoke API methods may take | ||
longer than expected or desired. By default, such methods all time out | ||
after a default interval, 60.0 seconds. Rather than blocking your application | ||
code for that interval, you may choose to configure explicit timeouts | ||
in your code, using one of three forms: | ||
|
||
- You can pass a single integer or float which functions as the timeout for the | ||
entire request. E.g.: | ||
|
||
.. code-block:: python | ||
bucket = client.get_bucket(BUCKET_NAME, timeout=300.0) # five minutes | ||
- You can also be passed as a two-tuple, ``(connect_timeout, read_timeout)``, | ||
where the ``connect_timeout`` sets the maximum time required to establish | ||
the connection to the server, and the ``read_timeout`` sets the maximum | ||
time to wait for a completed response. E.g.: | ||
|
||
.. code-block:: python | ||
bucket = client.get_bucket(BUCKET_NAME, timeout=(3, 10)) | ||
- You can also pass ``None`` as the timeout value: in this case, the library | ||
will block indefinitely for a response. E.g.: | ||
|
||
.. code-block:: python | ||
bucket = client.get_bucket(BUCKET_NAME, timeout=None) | ||
.. note:: | ||
Depending on the retry strategy, a request may be | ||
repeated several times using the same timeout each time. | ||
|
||
See also: | ||
|
||
:ref:`Timeouts in requests <requests:timeouts>` | ||
|
||
|
||
.. _configuring_retries: | ||
|
||
Configuring Retries | ||
-------------------- | ||
|
||
.. note:: | ||
|
||
For more background on retries, see also the | ||
`GCS Retry Strategies Document <https://cloud.google.com/storage/docs/retry-strategy#python>`_ | ||
|
||
Methods which invoke API methods may fail for a number of reasons, some of | ||
which represent "transient" conditions, and thus can be retried | ||
automatically. The library tries to provide a sensible default retry policy | ||
for each method, base on its semantics: | ||
|
||
- For API requests which are always idempotent, the library uses its | ||
:data:`~google.cloud.storage.retry.DEFAULT_RETRY` policy, which | ||
retries any API request which returns a "transient" error. | ||
|
||
- For API requests which are idempotent only if the blob has | ||
the same "generation", the library uses its | ||
:data:`~google.cloud.storage.retry.DEFAULT_RETRY_IF_GENERATION_SPECIFIED` | ||
policy, which retries API requests which returns a "transient" error, | ||
but only if the original request includes an ``ifGenerationMatch`` header. | ||
|
||
- For API requests which are idempotent only if the bucket or blob has | ||
the same "metageneration", the library uses its | ||
:data:`~google.cloud.storage.retry.DEFAULT_RETRY_IF_METAGENERATION_SPECIFIED` | ||
policy, which retries API requests which returns a "transient" error, | ||
but only if the original request includes an ``ifMetagenerationMatch`` header. | ||
|
||
- For API requests which are idempotent only if the bucket or blob has | ||
the same "etag", the library uses its | ||
:data:`~google.cloud.storage.retry.DEFAULT_RETRY_IF_ETAG_IN_JSON` | ||
policy, which retries API requests which returns a "transient" error, | ||
but only if the original request includes an ``ETAG`` in its payload. | ||
|
||
- For those API requests which are never idempotent, the library passes | ||
``retry=None`` by default, suppressing any retries. | ||
|
||
Rather than using one of the default policies, you may choose to configure an | ||
explicit policy in your code. | ||
|
||
- You can pass ``None`` as a retry policy to disable retries. E.g.: | ||
|
||
.. code-block:: python | ||
bucket = client.get_bucket(BUCKET_NAME, retry=None) | ||
- You can pass an instance of :class:`google.api_core.retry.Retry` to enable | ||
retries; the passed object will define retriable response codes and errors, | ||
as well as configuring backoff and retry interval options. E.g.: | ||
|
||
.. code-block:: python | ||
from google.api_core import exceptions | ||
from google.api_core.retry import Retry | ||
_MY_RETRIABLE_TYPES = [ | ||
exceptions.TooManyRequests, # 429 | ||
exceptions.InternalServerError, # 500 | ||
exceptions.BadGateway, # 502 | ||
exceptions.ServiceUnavailable, # 503 | ||
] | ||
def is_retryable(exc): | ||
return isinstance(exc, _MY_RETRIABLE_TYPES) | ||
my_retry_policy = Retry(predicate=is_retryable) | ||
bucket = client.get_bucket(BUCKET_NAME, retry=my_retry_policy) | ||
- You can pass an instance of | ||
:class:`google.cloud.storage.retry.ConditionalRetryPolicy`, which wraps a | ||
:class:`~google.cloud.storage.retry.RetryPolicy`, activating it only if | ||
certain conditions are met. This class exists to provide safe defaults | ||
for RPC calls that are not technically safe to retry normally (due to | ||
potential data duplication or other side-effects) but become safe to retry | ||
if a condition such as if_metageneration_match is set. E.g.: | ||
|
||
.. code-block:: python | ||
from google.api_core.retry import Retry | ||
from google.cloud.storage.retry import ConditionalRetryPolicy | ||
from google.cloud.storage.retry import is_etag_in_json | ||
def is_retryable(exc): | ||
... # as above | ||
my_retry_policy = Retry(predicate=is_retryable) | ||
my_cond_policy = ConditionalRetryPolicy( | ||
my_retry_policy, conditional_predicate=is_etag_in_json) | ||
bucket = client.get_bucket(BUCKET_NAME, retry=my_cond_policy) | ||
Retry Module API | ||
---------------- | ||
|
||
.. automodule:: google.cloud.storage.retry | ||
:members: | ||
:show-inheritance: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.