Various Intermittent Request Errors #3845
Comments
|
So the first four errors all amount to "the connection was closed by the remote peer at a time we didn't expect". The fifth one is a standard connection timeout. While it's possible that some of these reflect underlying problems (for example, a connection closed during a TLS handshake usually indicates that your TLS configuration is not supported by the server), in general all of these can be lumped together as what I'd call "network noise". Essentially, when using keep-alive connections it is possible for all kinds of wacky stuff to go on at the connection management layer. The simplest way to handle this, and something I highly recommend, is to turn on retries. This can be done at the transport adapter level like this: import requests
from requests.adapters import HTTPAdapter
s = requests.Session()
s.mount('http', HTTPAdapter(max_retries=3))
s.mount('https', HTTPAdapter(max_retries=3))If you want even more control you can pass a urllib3 However, without more specificity we can't help you with the individual problems. If you can get down to single reproducing cases that'd be fab. |
|
The code doesn't seem to handle this correctly. Reference: https://github.com/kennethreitz/requests/blob/master/requests/adapters.py#L110 I'll try to set it on the adapter after initialisation with: retries = Retry(
total=gitl.app.config['MAX_REQUEST_SESSION_RETRIES'],
backoff_factor=0.1,
status_forcelist=[500, 502, 503, 504],
)
adapter = HTTPAdapter()
adapter.max_retries = retries
self.session.mount('http', adapter)
self.session.mount('https', adapter)
It seems to be passes through correctly. However I still get the errors. Very odd... Edit: Turns out |
|
Indeed, this is because POST requests are not idempotent so it is not safe to automatically retry all POST requests. If you know POST requests are safe in this case, you can use urllib3's detailed Retry configuration object to allow retries to POST requests. |
|
Yes, that makes perfect sense. This now seems to be working fine. I'm just wondering how I would urllib3 retry to also handle |
|
Hrm, do you have the full trace back? I suspect that's a bug. |
|
Kk, I'll try to get more information the next time it happens! |
|
Here is the full stack trace: |
|
This is one of those moments where I'd have liked to see more data out of the stack trace. I swear, Python 3's chained exceptions only show up in my life when they aren't helpful, and never when they would be. 😉 However, I think this is a real bug. Looking at urllib3's code, it simply does not consider SSL errors to be worth retrying on. I think that instinct is wrong: SSL errors are almost certainly worth a retry attempt. I recommend you raise this as a bug over on the urllib3 repository, where we'll deal with it. For when you raise the bug, please attach the following repro code. Specifically, if the following server is run: import time
import socket
s = socket.socket()
s.bind(('', 4433))
s.listen(10)
while True:
news, _ = s.accept()
time.sleep(0.5)
news.close()Then this code will exhibit the issue: import urllib3
http = urllib3.PoolManager()
http.request('GET', 'https://localhost:4433/', retries=5)That code should fire off a |
|
For anyone reading this: Forking python processes doesn't seem to play nicely with requests. Ideally you have a single worker thread that is shared among all processes. This seems to have resolved all our errors here. |
|
I've noticed a pattern of receiving |
|
When using requests.post (have not tried other methods) I get the following errors intermittently.
Errors
The amount this happens differs, but at least a few times every hour (we are sending a few thousand logs per hour). I wasn't able to reproduce these errors locally or reliably.
I have experienced this will various targets hosts, namely the api provided by: loggly.com, logentries.com, logz.io. Since they all have the issue I don't think it is a server configuration problem.
This issue first appeared about a year ago when we set this up. However it seems to have gotten more problematic since we are sending more requests.
Setup
Running Python on Elastic Beanstalk inside a Single Container Docker Configuration behind an Elastic Load Balancer. The outgoing requests happen in various threads (not just in the main thread).
More information that might be useful:
cat /etc/*-release)docker --version)cat /etc/*-release)pip freeze --localresults:alembic==0.8.10
aniso8601==1.2.0
apns==2.0.1
awsebcli==3.9.0
backports.ssl-match-hostname==3.5.0.1
blessed==1.9.5
boto==2.45.0
botocore==1.5.7
cement==2.8.2
cffi==1.9.1
click==6.7
colorama==0.3.7
contextlib2==0.5.4
coverage==4.3.4
cryptography==1.7.2
docker-py==1.7.2
dockerpty==0.4.1
docopt==0.4.0
docutils==0.13.1
enum34==1.1.6
Faker==0.7.7
Flask==0.12
Flask-Compress==1.4.0
Flask-Limiter==0.9.3
Flask-Migrate==2.0.3
Flask-RESTful==0.3.5
-e flask-restful-swagger==0.15.5
Flask-Script==2.0.5
Flask-SQLAlchemy==2.1
Flask-Testing==0.6.1
funcsigs==1.0.2
GeoAlchemy2==0.4.0
idna==2.2
ipaddress==1.0.18
itsdangerous==0.24
Jinja2==2.9.5
jmespath==0.9.1
limits==1.2.1
Mako==1.0.6
mandrill==1.0.57
MarkupSafe==0.23
mock==2.0.0
nose==1.3.7
pathspec==0.5.0
pbr==1.10.0
pep8==1.7.0
pexpect==4.2.1
psycopg2==2.6.2
ptyprocess==0.5.1
pyasn1==0.1.9
pybars3==0.9.2
pycparser==2.17
pycrypto==2.6.1
pymemcache==1.4.0
PyMeta3==0.5.1
pyOpenSSL==16.2.0
python-dateutil==2.6.0
python-editor==1.0.3
python-gcm==0.1.5
pytz==2016.10
PyYAML==3.12
requests==2.13.0
rollbar==0.13.10
semantic-version==2.5.0
six==1.10.0
SQLAlchemy==1.1.5
tabulate==0.7.5
uWSGI==2.0.14
vcrpy==1.10.5
wcwidth==0.1.7
websocket-client==0.40.0
Werkzeug==0.11.15
wrapt==1.10.8
Not working Solutions
I have found similar issues here, but was not able to fix it with any of the described solutions. I have tried a lot of different approaches, non of which seem to work. Would much appreciate advice and what I should try next.
from requests.packages.urllib3.contrib import pyopenssl; pyopenssl.inject_into_urllib3()like described here.What else should I try? Any idea what might be causing this? What tools should I use to debug this? I'm kind of at a loss here.
The text was updated successfully, but these errors were encountered: