I'm currently getting the error, socket.error: [Errno 54] Connection reset by peer when I use requests.get(url, params=kwargs) and the params contain large bodies of text. If I truncate the two large bodies of text to less than 2,900 characters each, it works. If I run the same get request from the command line using curl it works. I'm using requests version 0.6.1 that I installed using, pip install python-requests.
socket.error: [Errno 54] Connection reset by peer
pip install python-requests
I'm not sure how to tell you to replicate the issue because I'm using the python-sendgrid library to add a newsletter to my sendgrid account and I don't want to post my api username and password in issue ticket. :)
To test from the command line using curl I created two files that each contained plain text and html text that was urlencoded using a urlencoder. Then I ran the following command.
export IDENTITY='<my identity number>'
export API_KEY='<my smtp password>'
export API_USER='<my smtp username>'
export NAME='<My Urlencoded Newsletter Name>'
export SUBJECT='<My Urlencoded Newsletter Subject>'
TEXT=`cat urlencoded.txt`; HTML=`cat urlencoded.html`; curl -d "api_user=$API_USER&api_key=$API_KEY&identity=$IDENTITY&name=$NAME&subject=$SUBJECT&text=$TEXT&html=$HTML" https://sendgrid.com/api/newsletter/newsletter/add.json
Hmm, interesting. If that's the error that is being raised, I have little reason to believe that the server isn't actually resetting the connection.
Do you have an HTTP proxy like Charles available? It might help shed some light onto what's actually going on.
Sorry, I forgot to provide the full traceback. Here is the traceback I get when I get the error. Also, I'm not understanding how to use the HTTP proxy and why that would make a difference when it works from the command line with curl.
Traceback (most recent call last):
File "/Users/oconnor/.virtualenvs/emails/bin/django-admin.py", line 5, in <module>
File "/Users/oconnor/.virtualenvs/emails/lib/python2.7/site-packages/django/core/management/__init__.py", line 429, in execute_from_command_line
File "/Users/oconnor/.virtualenvs/emails/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
File "/Users/oconnor/.virtualenvs/emails/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
File "/Users/oconnor/.virtualenvs/emails/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
output = self.handle(*args, **options)
File "/Users/oconnor/Sites/wenworld/emails/emails/management/commands/ww_daily_headlines_send.py", line 114, in handle
File "/Users/oconnor/Sites/wenworld/emails/emails/management/commands/ww_daily_headlines_send.py", line 94, in add_newsletter_to_sendgrid
File "/Users/oconnor/.virtualenvs/emails/src/sendgrid/src/sendgrid/__init__.py", line 67, in newsletter_add
subject=subject, text=text, html=html)
File "/Users/oconnor/.virtualenvs/emails/src/sendgrid/src/sendgrid/__init__.py", line 157, in get
return self.call(method, **kwargs)
File "/Users/oconnor/.virtualenvs/emails/src/sendgrid/src/sendgrid/__init__.py", line 56, in call
result_json = json.loads(response.content)
File "/Users/oconnor/.virtualenvs/emails/lib/python2.7/site-packages/requests/models.py", line 429, in __getattr__
self._content = self.read()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 351, in read
data = self._sock.recv(rbufsize)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 553, in read
s = self.fp.read(amt)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1276, in read
return s + self._file.read(amt - len(s))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 219, in recv
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 138, in read
socket.error: [Errno 54] Connection reset by peer
The response I get back from github is...
<head><title>414 Request-URI Too Large</title></head>
<center><h1>414 Request-URI Too Large</h1></center>
Which is strange because, like I mentioned before it works with curl.
If you take the time to take a look at the two requests through Charles (or similar), I'm sure you'll see the issue quite quickly.
As the error says, the Request-URI is too large for the server to process, so it's dropping the connection.
With Requests, you're sending all the data as a massive query string in the Request URI, known as a GET parameter. When you're using curl, you're uploading form-encoded data to the body of the request, known as POST data.
I've tried using both requests.get(url, params=kwargs) and requests.post(url, params=kwargs) and both return the same error. For some reason Charles isn't capturing requests from curl and it's just capturing requests from my browser and my python command lines script that uses python-requests.
params is for query strings, data is for POST data.
Sweet action! Switching from requests.get(url, params=kwargs) to requests.post(url, data=kwargs) fixed it! Sorry for bother you with this and thanks for the help!
No worries at all :)
It's strange that I get the same error with similar data, the only difference is that in my case I do not have big list or size of parameters. The same request made via CURL works fine but via requests gives me this error.
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='sbarnea.com', port=443): Max retries exceeded with url: /jira/rest/api/2/group?groupname=jira-administrators&expand=users (Caused by : [Errno 54] Connection reset by peer)
Even more interesting, it seems that the request does not even reach the web server.
Any idea what it could be cause?
@ssbarnea This issue is more than two years old: it has nothing to do with the problem you're currently experiencing. =) Can you open a new issue to track your problem please?
Even better, don't open an issue, go talk to StackOverflow, you're asking a question and you have no reason to believe it is an issue with how requests behaves. Hash out the details on StackOverflow and then we can file a bug report (if necessary).
It's a very strange behavior and I haver reasons to believe it's not a bug in requests, that's why I did open a new bug. I'm still investigating what could cause this, it happens only from my machine, but the most interesting part is that CURL works (and browsers too). Thank you.
Sorry to keep this here but I found what cause this:
# resolver 127.0.0.1;
# ssl_protocols TLSv1.2 TLSv1.1 ; # TLSv1
# ssl_stapling on;
# ssl_stapling_verify on;
# ssl_session_cache builtin:1000;
# ssl_session_timeout 30m;
# ssl_ciphers HIGH:!RC4:!aNULL:!MD5;
# ssl_prefer_server_ciphers on;
By disabling these tunnings on the nginx server it started to work with requests, clearly one of them is causing the malfunction.
It most probably a buggy server doing wrong stuff when spoken to in specific SSL/TLS versions like described in #1567 (and several more).
Try to force a different version like it is explained here (some more kudos to @Lukasa)
And the winner (losser) is ssl_protocols TLSv1.2 TLSv1.1;
It seems that removing TLSv1 from the supported protocols on the server prevent requests from working.
ssl_protocols TLSv1.2 TLSv1.1;
Now, I can call this a bug in requests or an underlying library?
Note: removal of TLSv1 is recommended due to security concerns, there are exploits with it.
@ssbarnea You'll find that the limitation isn't in Requests but in Python. Currently no shipped versions of Python have support for any version of TLS more recent than V1.0 as you can see in the table here.
Python 3.4 (not yet shipped, no guarantees that Requests is compatible with it) allows support for TLS1.2 and TLS1.1, so you can try with that. Otherwise, you'll have to allow TLSv1 to allow Requests through.
Still there is something strange: in both cases I used Python, when tested from OS X it failed, when tested from Ubuntu it worked.
Normally I wouldn't touch the default settings for SSL, but I run a SSL report and tried to solve most warnings:
FIPS requires: SSL v2 and SSL v3 are disabled (only TLS version protocols in use)
Also read about BEAST attacks.
Check it here:
So, I wondering if it's possible to fix this, clearly if it works with Python on Ubuntu, it cannot be specific to all python versions.
Mm, sadly it can. The relative features of the ssl support you get depends on the version of OpenSSL that you have installed. Compare the two versions. =)
Just to give a bit of closure on the subject. If your webserver doesn't allow TLSv1 then you will get TLSv1.1 and 1.2 with python 2.7.9 (due out in december 2014). However you can also install PYOpenSSL and inject that into urrlib3 (http://urllib3.readthedocs.org/en/latest/security.html#openssl-pyopenssl) which seems to allow use of requests for a webserver that has disabled TLSv1 with python 2.7.6 (and probably others).
But really, the webserver should probably not be deprecating something which forces people to jump through this many weird hoops.