socket.error: [Errno 54] Connection reset by peer #171

Closed
epicserve opened this Issue Sep 22, 2011 · 19 comments

7 participants

@epicserve

I'm currently getting the error, socket.error: [Errno 54] Connection reset by peer when I use requests.get(url, params=kwargs) and the params contain large bodies of text. If I truncate the two large bodies of text to less than 2,900 characters each, it works. If I run the same get request from the command line using curl it works. I'm using requests version 0.6.1 that I installed using, pip install python-requests.

I'm not sure how to tell you to replicate the issue because I'm using the python-sendgrid library to add a newsletter to my sendgrid account and I don't want to post my api username and password in issue ticket. :)

To test from the command line using curl I created two files that each contained plain text and html text that was urlencoded using a urlencoder. Then I ran the following command.

export IDENTITY='<my identity number>'
export API_KEY='<my smtp password>'
export API_USER='<my smtp username>'
export NAME='<My Urlencoded Newsletter Name>'
export SUBJECT='<My Urlencoded Newsletter Subject>'
TEXT=`cat urlencoded.txt`; HTML=`cat urlencoded.html`; curl -d "api_user=$API_USER&api_key=$API_KEY&identity=$IDENTITY&name=$NAME&subject=$SUBJECT&text=$TEXT&html=$HTML" https://sendgrid.com/api/newsletter/newsletter/add.json
@kennethreitz

Hmm, interesting. If that's the error that is being raised, I have little reason to believe that the server isn't actually resetting the connection.

Do you have an HTTP proxy like Charles available? It might help shed some light onto what's actually going on.

@epicserve

Sorry, I forgot to provide the full traceback. Here is the traceback I get when I get the error. Also, I'm not understanding how to use the HTTP proxy and why that would make a difference when it works from the command line with curl.

Traceback (most recent call last):
  File "/Users/oconnor/.virtualenvs/emails/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
  File "/Users/oconnor/.virtualenvs/emails/lib/python2.7/site-packages/django/core/management/__init__.py", line 429, in execute_from_command_line
    utility.execute()
  File "/Users/oconnor/.virtualenvs/emails/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/oconnor/.virtualenvs/emails/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/oconnor/.virtualenvs/emails/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/Users/oconnor/Sites/wenworld/emails/emails/management/commands/ww_daily_headlines_send.py", line 114, in handle
    self.add_newsletter_to_sendgrid(sendgrid_newsletter_name, NEWSLETTER_SLUG)
  File "/Users/oconnor/Sites/wenworld/emails/emails/management/commands/ww_daily_headlines_send.py", line 94, in add_newsletter_to_sendgrid
    html=email.html
  File "/Users/oconnor/.virtualenvs/emails/src/sendgrid/src/sendgrid/__init__.py", line 67, in newsletter_add
    subject=subject, text=text, html=html)
  File "/Users/oconnor/.virtualenvs/emails/src/sendgrid/src/sendgrid/__init__.py", line 157, in get
    return self.call(method, **kwargs)
  File "/Users/oconnor/.virtualenvs/emails/src/sendgrid/src/sendgrid/__init__.py", line 56, in call
    result_json = json.loads(response.content)
  File "/Users/oconnor/.virtualenvs/emails/lib/python2.7/site-packages/requests/models.py", line 429, in __getattr__
    self._content = self.read()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 351, in read
    data = self._sock.recv(rbufsize)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 553, in read
    s = self.fp.read(amt)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1276, in read
    return s + self._file.read(amt - len(s))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 219, in recv
    return self.read(buflen)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 138, in read
    return self._sslobj.read(len)
socket.error: [Errno 54] Connection reset by peer
@epicserve

The response I get back from github is...

<html>
<head><title>414 Request-URI Too Large</title></head>
<body bgcolor="white">
<center><h1>414 Request-URI Too Large</h1></center>
<hr><center>nginx/0.7.65</center>
</body>
</html>

Which is strange because, like I mentioned before it works with curl.

@kennethreitz

If you take the time to take a look at the two requests through Charles (or similar), I'm sure you'll see the issue quite quickly.


As the error says, the Request-URI is too large for the server to process, so it's dropping the connection.

With Requests, you're sending all the data as a massive query string in the Request URI, known as a GET parameter. When you're using curl, you're uploading form-encoded data to the body of the request, known as POST data.

@epicserve

I've tried using both requests.get(url, params=kwargs) and requests.post(url, params=kwargs) and both return the same error. For some reason Charles isn't capturing requests from curl and it's just capturing requests from my browser and my python command lines script that uses python-requests.

@kennethreitz

No worries.

params is for query strings, data is for POST data.

@epicserve

Sweet action! Switching from requests.get(url, params=kwargs) to requests.post(url, data=kwargs) fixed it! Sorry for bother you with this and thanks for the help!

@epicserve epicserve closed this Sep 22, 2011
@kennethreitz

No worries at all :)

@ssbarnea

It's strange that I get the same error with similar data, the only difference is that in my case I do not have big list or size of parameters. The same request made via CURL works fine but via requests gives me this error.

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='sbarnea.com', port=443): Max retries exceeded with url: /jira/rest/api/2/group?groupname=jira-administrators&expand=users (Caused by : [Errno 54] Connection reset by peer)

Even more interesting, it seems that the request does not even reach the web server.

Any idea what it could be cause?

@Lukasa
Collaborator

@ssbarnea This issue is more than two years old: it has nothing to do with the problem you're currently experiencing. =) Can you open a new issue to track your problem please?

@sigmavirus24
Collaborator

Even better, don't open an issue, go talk to StackOverflow, you're asking a question and you have no reason to believe it is an issue with how requests behaves. Hash out the details on StackOverflow and then we can file a bug report (if necessary).

@ssbarnea

It's a very strange behavior and I haver reasons to believe it's not a bug in requests, that's why I did open a new bug. I'm still investigating what could cause this, it happens only from my machine, but the most interesting part is that CURL works (and browsers too). Thank you.

@ssbarnea

Sorry to keep this here but I found what cause this:

#  resolver 127.0.0.1;
#  ssl_protocols TLSv1.2 TLSv1.1 ; # TLSv1
#  ssl_stapling on;
#  ssl_stapling_verify on;
#  ssl_session_cache builtin:1000;
#  ssl_session_timeout 30m;
#  ssl_ciphers HIGH:!RC4:!aNULL:!MD5;
#  ssl_prefer_server_ciphers on;

By disabling these tunnings on the nginx server it started to work with requests, clearly one of them is causing the malfunction.

@t-8ch

@ssbarnea
It most probably a buggy server doing wrong stuff when spoken to in specific SSL/TLS versions like described in #1567 (and several more).
Try to force a different version like it is explained here (some more kudos to @Lukasa)

@ssbarnea

And the winner (losser) is ssl_protocols TLSv1.2 TLSv1.1;
It seems that removing TLSv1 from the supported protocols on the server prevent requests from working.

Now, I can call this a bug in requests or an underlying library?

Note: removal of TLSv1 is recommended due to security concerns, there are exploits with it.

@Lukasa
Collaborator

@ssbarnea You'll find that the limitation isn't in Requests but in Python. Currently no shipped versions of Python have support for any version of TLS more recent than V1.0 as you can see in the table here.

Python 3.4 (not yet shipped, no guarantees that Requests is compatible with it) allows support for TLS1.2 and TLS1.1, so you can try with that. Otherwise, you'll have to allow TLSv1 to allow Requests through.

@ssbarnea

Still there is something strange: in both cases I used Python, when tested from OS X it failed, when tested from Ubuntu it worked.

Normally I wouldn't touch the default settings for SSL, but I run a SSL report and tried to solve most warnings:

FIPS requires: SSL v2 and SSL v3 are disabled (only TLS version protocols in use)
Also read about BEAST attacks.

Check it here:
https://sslcheck.globalsign.com/en_US/sslcheck/?host=sbarnea.com

So, I wondering if it's possible to fix this, clearly if it works with Python on Ubuntu, it cannot be specific to all python versions.

@Lukasa
Collaborator

Mm, sadly it can. The relative features of the ssl support you get depends on the version of OpenSSL that you have installed. Compare the two versions. =)

@chrisdl

Just to give a bit of closure on the subject. If your webserver doesn't allow TLSv1 then you will get TLSv1.1 and 1.2 with python 2.7.9 (due out in december 2014). However you can also install PYOpenSSL and inject that into urrlib3 (http://urllib3.readthedocs.org/en/latest/security.html#openssl-pyopenssl) which seems to allow use of requests for a webserver that has disabled TLSv1 with python 2.7.6 (and probably others).

import urllib3.contrib.pyopenssl
urllib3.contrib.pyopenssl.inject_into_urllib3()

But really, the webserver should probably not be deprecating something which forces people to jump through this many weird hoops.

@cyberb cyberb referenced this issue in syncloud/platform Nov 8, 2015
Open

Enforce TLS v1.2 #74

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment