Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Retry on API failure when connecting from builders #4902

Merged
merged 2 commits into from
Nov 19, 2018
Merged

Retry on API failure when connecting from builders #4902

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
eee2c36
Retry mechanism for builders to keep trying when webs are overloaded
humitos Nov 14, 2018
3d8ac2e
NOQA on import that lint doesn't find
humitos Nov 19, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion readthedocs/restapi/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@

import requests
from django.conf import settings
from requests.packages.urllib3.util.retry import Retry # noqa
from requests_toolbelt.adapters import host_header_ssl
from rest_framework.renderers import JSONRenderer
from slumber import API, serialize
Expand Down Expand Up @@ -44,9 +45,21 @@ def setup_api():
else:
adapter_class = requests.adapters.HTTPAdapter

# Define a retry mechanism trying to attempt to not fail in the first
# error. Builders hit this issue frequently because the webs are high loaded
retry = Retry(
total=3,
read=3,
connect=3,
status=3,
backoff_factor=0.5, # 0.5, 1, 2 seconds
method_whitelist=('GET', 'PUT', 'PATCH', 'POST'),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

POST could be a little dangerous here, but considering that we are using it to create a new BuildCommand I think it's necessary.

https://github.com/rtfd/readthedocs.org/blob/d9a9af14260fcf9ffcbb63cd0b986d7f1a83079a/readthedocs/doc_builder/environments.py#L227

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the docs

By default, we only retry on methods which are considered to be idempotent (multiple requests with the same parameters end with the same state). See Retry.DEFAULT_METHOD_WHITELIST.

https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.retry.Retry.DEFAULT_METHOD_WHITELIST

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be a little concerned about the POST retry as well, but these are the calls that we're currently failing on, so I'm not sure if there is anything we want to do differently here.

status_forcelist=(408, 413, 429, 500, 502, 503, 504),
)

session.mount(
API_HOST,
adapter_class(max_retries=3),
adapter_class(max_retries=retry),
)
session.headers.update({'Host': PRODUCTION_DOMAIN})
api_config = {
Expand Down