New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Retry batch requests (fixes #39) #51
Changes from 7 commits
e27d484
bb64f40
ef41e40
9f95a33
8b7012a
3fdeb30
a647998
42faf09
14a6640
613d1ec
a50cde4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -211,12 +211,39 @@ It is possible to do batch requests using a Python context manager (``with``): | |
.. code-block:: python | ||
|
||
with client.batch() as batch: | ||
for idx in range(0,100): | ||
for idx in range(0,100): | ||
batch.update_record(data={'id': idx}) | ||
|
||
A batch object shares the same methods as another client. | ||
|
||
|
||
Retry on error | ||
-------------- | ||
|
||
When the server is throttled, under heavy load, or maintenance, it can | ||
return error responses. | ||
|
||
The client can hence retry to send the same request until it succeeds. | ||
To enable this, specify the number of retries on the client: | ||
|
||
.. code-block:: python | ||
|
||
client = Client(server_url='http://localhost:8888/v1', | ||
auth=credentials, | ||
retry=10) | ||
|
||
In the Kinto protocol, it is specified that the server `tells the duration in seconds between retries | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "The Kinto protocol let the server define the duration between retries" |
||
<http://kinto.readthedocs.org/en/latest/api/1.x/cliquet/backoff.html#retry-after-indicators>`_. | ||
It is possible to force this value: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is possible (but not recommended) to force this value in the client: |
||
|
||
.. code-block:: python | ||
|
||
client = Client(server_url='http://localhost:8888/v1', | ||
auth=credentials, | ||
retry=10, | ||
retry_after=5) | ||
|
||
|
||
Run tests | ||
========= | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,95 @@ | ||
import time | ||
|
||
import requests | ||
from six.moves.urllib.parse import urlparse | ||
|
||
from kinto_client import utils | ||
from kinto_client.exceptions import KintoException | ||
|
||
|
||
def create_session(server_url=None, auth=None, session=None, retry=0, | ||
retry_after=None): | ||
"""Returns a session from the passed arguments. | ||
|
||
:param server_url: | ||
The URL of the server to use, with the prefix. | ||
:param auth: | ||
A requests authentication policy object. | ||
:param session: | ||
An optional session object to use, rather than creating a new one. | ||
""" | ||
# XXX Refactor the create_session to take place in the caller objects. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One of the purposes of this function was to return the session, rather than doing the check in all the callers. Seeing this note, I wonder what's the best approach. |
||
# E.g. test if the session exists before calling create_session. | ||
if session is not None and ( | ||
server_url is not None or auth is not None): | ||
msg = ("You cannot specify session and server_url or auth. " | ||
"Chose either session or (auth + server_url).") | ||
raise AttributeError(msg) | ||
if session is None and server_url is None and auth is None: | ||
msg = ("You need to either set session or auth + server_url") | ||
raise AttributeError(msg) | ||
if session is None: | ||
session = Session(server_url=server_url, auth=auth, retry=retry, | ||
retry_after=retry_after) | ||
return session | ||
|
||
|
||
class Session(object): | ||
"""Handles all the interactions with the network. | ||
""" | ||
def __init__(self, server_url, auth=None, retry=0, retry_after=None): | ||
self.server_url = server_url | ||
self.auth = auth | ||
self.nb_retry = retry | ||
self.retry_after = retry_after | ||
|
||
def request(self, method, endpoint, data=None, permissions=None, | ||
payload=None, **kwargs): | ||
parsed = urlparse(endpoint) | ||
if not parsed.scheme: | ||
actual_url = utils.urljoin(self.server_url, endpoint) | ||
else: | ||
actual_url = endpoint | ||
|
||
if self.auth is not None: | ||
kwargs.setdefault('auth', self.auth) | ||
|
||
payload = payload or {} | ||
# if data is not None: | ||
payload['data'] = data or {} | ||
if permissions is not None: | ||
if hasattr(permissions, 'as_dict'): | ||
permissions = permissions.as_dict() | ||
payload['permissions'] = permissions | ||
if payload: | ||
payload_kwarg = 'data' if 'files' in kwargs else 'json' | ||
kwargs.setdefault(payload_kwarg, payload) | ||
|
||
retry = self.nb_retry | ||
while retry >= 0: | ||
resp = requests.request(method, actual_url, **kwargs) | ||
retry = retry - 1 | ||
if not (200 <= resp.status_code < 400): | ||
if resp.status_code >= 500 and retry >= 0: | ||
# Wait and try again. | ||
# If not forced, use retry-after header and wait. | ||
if self.retry_after is None: | ||
retry_after = resp.headers.get("Retry-After", 0) | ||
else: | ||
retry_after = self.retry_after | ||
time.sleep(retry_after) | ||
continue | ||
|
||
# Retries exhausted, raise expection. | ||
message = '{0} - {1}'.format(resp.status_code, resp.json()) | ||
exception = KintoException(message) | ||
exception.request = resp.request | ||
exception.response = resp | ||
raise exception | ||
|
||
if resp.status_code == 304: | ||
body = None | ||
else: | ||
body = resp.json() | ||
# XXX Add the status code. | ||
return body, resp.headers |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
consider
When the server is throttled (under heavy load or maintenance), it can return error responses.