Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeError: getresponse() got an unexpected keyword argument 'buffering' #1915

Closed
jcea opened this issue Feb 13, 2014 · 36 comments
Closed

TypeError: getresponse() got an unexpected keyword argument 'buffering' #1915

jcea opened this issue Feb 13, 2014 · 36 comments

Comments

@jcea
Copy link

jcea commented Feb 13, 2014

Requests 2.2.1. Same thing happens in 1.2.3 (I upgraded from that).

I get this traceback:

Traceback (most recent call last):
  File "/usr/local/lib/python3.3/site-packages/requests/packages/urllib3/connectionpool.py", line 313, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.3/site-packages/requests/packages/urllib3/connectionpool.py", line 480, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python3.3/site-packages/requests/packages/urllib3/connectionpool.py", line 315, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.3/http/client.py", line 1147, in getresponse
    response.begin()
  File "/usr/local/lib/python3.3/http/client.py", line 358, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.3/http/client.py", line 328, in _read_status
    raise BadStatusLine(line)
http.client.BadStatusLine: ''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.3/site-packages/requests/adapters.py", line 330, in send
    timeout=timeout
  File "/usr/local/lib/python3.3/site-packages/requests/packages/urllib3/connectionpool.py", line 530, in urlopen
    raise MaxRetryError(self, url, e)
requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='heimdallr.jcea.es', port=443): Max retries exceeded with url: /PANICO (Caused by <class 'http.client.BadStatusLine'>: '')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./heimdallr.py", line 203, in <module>
    module.start()
  File "__main__", line 59, in start
  File "main", line 23, in start
  File "panic_report", line 17, in envia_tb_pendiente
  File "/usr/local/lib/python3.3/site-packages/requests/sessions.py", line 425, in post
    return self.request('POST', url, data=data, **kwargs)
  File "auth_http", line 48, in request
  File "/usr/local/lib/python3.3/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.3/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.3/site-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='heimdallr.jcea.es', port=443): Max retries exceeded with url: /PANICO (Caused by <class 'http.client.BadStatusLine'>: '')
Makefile:69: recipe for target 'run' failed
make: *** [run] Error 1
@jcea
Copy link
Author

jcea commented Feb 13, 2014

For my own reference: this is 100% reproductible in changeset "2e3cbe6aed98" in my "heimdallr" Mercurial project, when running on master Raspberry PI.

@sigmavirus24
Copy link
Contributor

@jcea did you search other issues on the project? Your ticket reminded me of https://github.com/kennethreitz/requests/issues/1289 but I searched for getresponse before I found it. The last activity on it was a month ago.

To update you, there are patches on bugs.python.org which haven't moved anywhere because no core developers have bothered reviewing them. If you want action on this, your best bet is to go find those issues on bugs.python.org and bump them. I have done that myself but perhaps the more people who do so, the faster a response.

Thanks for opening this, but it is a duplicate and it is not a bug in requests.

@Lukasa
Copy link
Member

Lukasa commented Feb 13, 2014

The relevant section of the linked issue is this one (from this comment):

The key is that the TypeError raised as the first exception is unrelated to the subsequent ones. In fact, that's the standard control flow in urllib3. This means that the real exception that's being raised here is therequest.exceptions.ConnectionError exception that wraps the urllib3.exceptions.MaxRetryError exception being raised in urllib3.

Please give that comment a read, it'll explain what you're actually seeing here.

@rramanadham
Copy link

Hi, I get the same issue when my url has the local IP on which we are running the code on. If the IP is not the local IP and a remote IP, it all works fine.

Error:

 File "/usr/lib/python3.3/site-packages/requests-2.3.0-py3.3.egg/requests/packages/urllib3/connectionpool.py", line 319, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./run.py", line 241, in <module>
    main()
  File "./run.py", line 47, in main
    updateDoc.main()
  File "/root/raman/db/src/srm/updateDoc.py", line 29, in main
    updateScore()
  File "/root/raman/db/src/srm/updateDoc.py", line 46, in updateScore
    response = requests.get(cpu_url)
  File "/usr/lib/python3.3/site-packages/requests-2.3.0-py3.3.egg/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python3.3/site-packages/requests-2.3.0-py3.3.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.3/site-packages/requests-2.3.0-py3.3.egg/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.3/site-packages/requests-2.3.0-py3.3.egg/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.3/site-packages/requests-2.3.0-py3.3.egg/requests/adapters.py", line 327, in send
    timeout=timeout
  File "/usr/lib/python3.3/site-packages/requests-2.3.0-py3.3.egg/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3.3/site-packages/requests-2.3.0-py3.3.egg/requests/packages/urllib3/connectionpool.py", line 321, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.3/http/client.py", line 1143, in getresponse
    response.begin()
  File "/usr/lib64/python3.3/http/client.py", line 354, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.3/http/client.py", line 316, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib64/python3.3/socket.py", line 297, in readinto
    return self._sock.recv_into(b)
KeyboardInterrupt

Here is my code -

__author__ = 'ramanr'

import cloudant
import requests
import math
import getInputs


database_name=''
test_id=''
db_uri=''
performance_view=''

inputs_dict = {}

def main():
    global db_uri, database_name, test_id,performance_view
    global avg_memory_threshold, avg_cpu_threshold, avg_fs_threshold
    global inputs_dict

    inputs_dict = getInputs.main()

    database_name = inputs_dict['database_name']
    test_id = inputs_dict['test_id']
    db_uri = inputs_dict['db_uri']
    performance_view = inputs_dict['performance_view']

    updateScore()

def updateScore():

    avg_memory_threshold = int(float(inputs_dict['avg_memory_threshold']))
    avg_cpu_threshold = int(float(inputs_dict['avg_cpu_threshold']))
    avg_fs_threshold = int(float(inputs_dict['avg_fs_threshold']))

    # cpu_url = "http://10.247.32.72:5984/longevity/_design/perfstats/_view/cpu?StorageResourceManagement_3.0SP1_test1"
    # memory_url = "http://10.247.32.72:5984/longevity/_design/perfstats/_view/memory?StorageResourceManagement_3.0SP1_test1"
    # filesystem_url = "http://10.247.32.72:5984/longevity/_design/perfstats/_view/filesystem?StorageResourceManagement_3.0SP1_test1"


    cpu_url = db_uri+'/'+database_name+'/'+'_design/'+performance_view+'/_view/'+'cpu?'+test_id
    memory_url = db_uri+'/'+database_name+'/'+'_design/'+performance_view+'/_view/'+'memory?'+test_id
    filesystem_url = db_uri+'/'+database_name+'/'+'_design/'+performance_view+'/_view/'+'filesystem?'+test_id

    response = requests.get(cpu_url)
    #print(response.json())
    dict = response.json()
    lists = dict['rows']
    cpu_score = getScore(lists, avg_cpu_threshold)

    response = requests.get(memory_url)
    dict = response.json()
    lists = dict['rows']
    memory_score = getScore(lists,avg_memory_threshold )

    response = requests.get(filesystem_url)
    dict = response.json()
    lists = dict['rows']
    filesystem_score = getScore(lists,avg_fs_threshold)

    score = round(math.ceil(((0.6*cpu_score)+(0.3*memory_score)+(0.1*filesystem_score))*100)/100, 1)
    print(cpu_score,memory_score,filesystem_score,score)
    updateDoc_score(db_uri, database_name, score)

def updateDoc_score(db_uri, database_name, score):
    db_server = cloudant.Account(uri=db_uri)
    db = db_server.database(database_name)
    #test_id = test_id
    doc = db.document(test_id)
    doc.merge(({'score': score}))



def getScore(lists, threshold):
    avg = math.ceil(lists[0]['value']*100)/100
    #IF(G3<80,10,10-((G3-80)/2))

    if avg < threshold:
        score = 10
    else:
        score = 10 - (avg - threshold)/2
    return(score)



#connect to couchDB and createDB (if necessory)
def updateDoc_errorCount():
    db_server = cloudant.Account(uri=db_uri)
    db = db_server.database(database_name)
    #test_id = test_id
    doc = db.document(test_id)
    errorcount = 100

    doc.merge(({'error_count': errorcount}))




if __name__ == "__main__":
   main()

So, I get the error on line 46(highlighted in the image below). If my url has the local IP, it doesnt work. If the url is has any other IP, it works. I am using python 3.3 and request 2.3.0

image

@sigmavirus24
Copy link
Contributor

@rramanadham I edited your comment to make it easier to read (using fenced code blocks).

The problem I'm having is that your traceback shows that the unhandled exception is a KeyboardInterrupt exception which means someone pressed Ctrl-C (assuming *nix, otherwise Ctrl-Z if I'm remembering my Windows correctly) which caused the traceback you're seeing. The problem is not passing buffering=True, that exception was handled. While it was being handled, someone killed the process.

If you can get me the actual Traceback we can look at this again. If that traceback shows something different, I'd suggest we open a new issues but only if we determine this isn't someone killing the process.

@rramanadham
Copy link

I am killing the process because if I dont kill it, it hangs forever.
But, my code runs smooth when I just have a remote IP in the url rather than the local IP.

@sigmavirus24
Copy link
Contributor

Right so this seems to be a different issue totally unrelated to this one you've commented on. What is db_uri in the local case?

@rramanadham
Copy link

db_uri = http://10.247.32.72:5984

@sigmavirus24
Copy link
Contributor

What happens when you curl the URL you generate?

@rramanadham
Copy link

nothing... i dont get response because the url is a couchdb view which runs based off documents that get created and my database has 0 documents. I don't know why.

lglod078:~ # curl -X GET 'http://10.247.32.78:5984/longevity/_design/perfstats/_view/cpu?SRMSuite_3.0.2_test1'

@Lukasa
Copy link
Member

Lukasa commented May 21, 2014

I'm confused: if you don't get a response when you use curl to get it, why would you expect one with requests?

@rramanadham
Copy link

You are right. I didn't know that the curl commands weren't working. The other ones worked, so assumed, this works too. Sorry to take your time. I was struggling for 2 days on this and during the conversation, figured that the problem is somewhere else. Sorry!

@Lukasa
Copy link
Member

Lukasa commented May 21, 2014

No need to apologise, I was just genuinely confused! You were quite right to have approached us, I just thought there was something I hadn't understood. =)

@awbacker
Copy link

awbacker commented Aug 3, 2014

Getting the same thing here. I am making a lot of queries to a JSON api, and this happens occasionally. Just did a batch of 100 with no issue. Before, it happened sometimes after around 20.

Running on os-x 10.9, installed via port for the base stuff, python 3.4.1, etc. I had to install a custom fork of something, i don't recall what, to get 3.x support. I don't think it had to do with this, though, probably ldap.

Traceback (most recent call last):
  File "/venv-root/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 319, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/venv-root/dpscrape/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
    body=body, headers=headers)
  File "/venv-root/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 321, in _make_request
    httplib_response = conn.getresponse()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1172, in getresponse
    response.begin()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 351, in begin
    version, status, reason = self._read_status()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 313, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 371, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 54] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/venv-root/lib/python3.4/site-packages/requests/adapters.py", line 327, in send
    timeout=timeout
  File "/venv-root/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 543, in urlopen
    raise MaxRetryError(self, url, e)
requests.packages.urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='server.com', port=80): Max retries exceeded with url: /some/v2/?key=abcde&city=&format=json (Caused by <class 'ConnectionResetError'>: [Errno 54] Connection reset by peer)

@sigmavirus24
Copy link
Contributor

@awbacker the real problem you're seeing has nothing to do with getresponse receiving an extra argument. What's causing this for you is that you're getting a ConnectionResetError while making a request.

@awbacker
Copy link

awbacker commented Aug 3, 2014

Okie. I figured I would get an error code of some kind and not an exception for that, and since it was in the middle I wasn't sure. Thanks for clearing that up, and forgive my newness at interpreting python stacktraces. Should I remove the comment and not muddy the issue?

@sigmavirus24
Copy link
Contributor

@awbacker when the connection is reset the only sane thing to do is throw an exception because the request/response cycle cannot be completed. Returning a Response object would be entirely disingenuous and there's no fake status code we could use to indicate anything other than a catastrophic failure.

forgive my newness at interpreting python stacktraces

No worries. Python 3's stacktraces seem to be a source of frustration for a large number of people. This issue continues to see activity because of that exact reason.

Should I remove the comment and not muddy the issue?

No need to remove it. If others find this and read the entire comment history, maybe it will give them a hint as to how to find the real problem they're encountering.

@piyushjajoo
Copy link

I'm getting a similar kind on error but with python3.4 library -

File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 319, in _make_request
httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 321, in _make_request
httplib_response = conn.getresponse()
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1172, in getresponse
response.begin()
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 351, in begin
version, status, reason = self._read_status()
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 313, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 371, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/adapters.py", line 327, in send
timeout=timeout
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
body=body, headers=headers)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 324, in _make_request
self, url, "Read timed out. (read timeout=%s)" % read_timeout)
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=8777): Read timed out. (read timeout=6)

@Lukasa
Copy link
Member

Lukasa commented Sep 6, 2014

@piyushjajoo No you aren't. Please read this issue carefully. I made this comment, which linked to this comment from issue #1289. Once again, reproducing the body of the comment:

The key is that the TypeError raised as the first exception is unrelated to the subsequent ones. In fact, that's the standard control flow in urllib3. This means that the real exception that's being raised here is the request.exceptions.ConnectionError exception that wraps the urllib3.exceptions.MaxRetryErrorexception being raised in urllib3.

In your case, the real exception you care about is the last one: the ReadTimeoutError. You should pursue that one.

@piyushjajoo
Copy link

Thanks, for pointing to correct issue.

@Gaurav-Ambasta
Copy link

@Lukasa : wondering if the issue is related , however i am trying different option as you stated earlier , but none seems to work for me , Please suggest if you have any pointer:

when I am making this api call to fetch the details from iCloud , it thows error

api.contacts.all()

Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 372, in _make_request
httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 374, in _make_request
httplib_response = conn.getresponse()
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1162, in getresponse
raise ResponseNotReady(self.__state)
http.client.ResponseNotReady: Request-sent

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 597, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/util/retry.py", line 245, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/packages/six.py", line 309, in reraise
raise value.with_traceback(tb)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 374, in _make_request
httplib_response = conn.getresponse()
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1162, in getresponse
raise ResponseNotReady(self.__state)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', ResponseNotReady('Request-sent',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/Users/gaurav/pyicloud-0.6.2/pyicloud/services/contacts.py", line 55, in all
self.refresh_client()
File "/Users/gaurav/pyicloud-0.6.2/pyicloud/services/contacts.py", line 44, in refresh_client
self.session1.post(self._contacts_changeset_url, params=params_refresh)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/sessions.py", line 508, in post
return self.request('POST', url, data=data, json=json, *_kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, *_send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ResponseNotReady('Request-sent',))

The code which is making this call is

host = self._service_root.split('//')[1].split(':')[0]
self.session.headers.update({'host': host})
params_contacts = dict(self.params)
params_contacts.update({
'clientVersion': '2.1',
'locale': 'en_US',
'order': 'last,first',
})
req = self.session.get(
self._contacts_refresh_url,
params=params_contacts
)
self.response = req.json()
params_refresh = dict(self.params)
params_refresh.update({
'prefToken': req.json()["prefToken"],
'syncToken': req.json()["syncToken"],
})
self.session.post(self._contacts_changeset_url, params=params_refresh)
req = self.session.get(
self._contacts_refresh_url,
params=params_contacts
)
self.response = req.json()

Appreciate your support . I am currently using Python 3.4.3 ..

@Lukasa
Copy link
Member

Lukasa commented Apr 28, 2015

This is not the same error, the error you're seeing is almost certainly #2568.

@Gaurav-Ambasta
Copy link

Thanks for your response.
Do we have any solution for this issue ? Also wondering if it is possible to resolve this with any other stable version of Python (say 2.7) .

@Lukasa
Copy link
Member

Lukasa commented Apr 29, 2015

As you can see on the discussion for #2568, we're working on it. in the meantime, you can resolve the problem by pinning to requests 2.6.0.

@1a1a11a
Copy link

1a1a11a commented May 4, 2015

I am new to python and requests, if I am doing anything wrong, please help me. Thanks!

I am trying to write a crawler to automatically download some files using requests module. However, I met a problem.

I initialized a new requests session, then I used post method to login into the website, after that as long as I try to use post/get method (a simplified code below):

s=requests.session()
s.post(url,data=post_data, headers=headers)
#up to here everything is correct, the next step will report error 
s.get(url) or s.post(url) even repeat s.post(url,data=post_data, headers=headers) will report error 

it will report error like the one below:

Traceback (most recent call last):
File"/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 372, in _make_request
httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 374, in _make_request
httplib_response = conn.getresponse()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1162, in getresponse
raise ResponseNotReady(self.__state)
http.client.ResponseNotReady: Request-sent

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 597, in urlopen
_stacktrace=sys.exc_info()[2])
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/util/retry.py", line 245, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/packages/six.py", line 309, in reraise
raise value.with_traceback(tb)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 374, in _make_request
httplib_response = conn.getresponse()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1162, in getresponse
raise ResponseNotReady(self.__state)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', ResponseNotReady('Request-sent',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "test.py", line 280, in <module>
test()
File "test.py", line 273, in test
emuch1.getEbook()
File "test.py", line 146, in getEbook
self.downloadEbook(ebook)
File "test.py", line 179, in downloadEbook
file_url=self.downloadEbookGetFileUrl(ebook).decode('gbk')
File "test.py", line 211, in downloadEbookGetFileUrl
download_url=self.downloadEbookGetUrl(ebook)
File "test.py", line 200, in downloadEbookGetUrl
respond_ebook=self.session.get(ebook_url)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/sessions.py", line 477, in get
return self.request('GET', url, **kwargs)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ResponseNotReady('Request-sent',))

I have totally no idea why this happens, can anyone help me?

@Lukasa
Copy link
Member

Lukasa commented May 4, 2015

@1a1a11a Try upgrading to requests 2.7.0 (released yesterday), which should fix your problem.

@1a1a11a
Copy link

1a1a11a commented May 4, 2015

@Lukasa YES! After upgrading, the problems seems to be solved!!! Thank you a lot!

@Lukasa
Copy link
Member

Lukasa commented May 4, 2015

@1a1a11a My pleasure. =)

@natebirkholz
Copy link

Thank heavens!! I just came here for this and it was fixed, also!!!

@krgough
Copy link

krgough commented May 6, 2015

Yay - fixed it for me

@hickford
Copy link

I had the same problem. Fixed by updating requests pip install --upgrade requests

@laike9m
Copy link

laike9m commented Jan 12, 2016

Having the same problem with Python 3.5.0 and requests 2.9.1 on OS X 10.11

Traceback is almost the same as @1a1a11a

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 376, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 378, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1174, in getresponse
    response.begin()
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 282, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 243, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 571, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 924, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 786, in read
    return self._sslobj.read(len, buffer)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 570, in read
    v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 54] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 609, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py", line 247, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/packages/six.py", line 309, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 378, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1174, in getresponse
    response.begin()
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 282, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 243, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 571, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 924, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 786, in read
    return self._sslobj.read(len, buffer)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 570, in read
    v = self._sslobj.read(len, buffer)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/laike9m/ICT/zhihu-analysis/dynamic/main.py", line 108, in <module>
    main()
  File "/Users/laike9m/ICT/zhihu-analysis/dynamic/main.py", line 89, in main
    m.detect_new_question()
  File "/Users/laike9m/ICT/zhihu-analysis/dynamic/monitor.py", line 32, in detect_new_question
    question = latest_question = next(it)
  File "/usr/local/lib/python3.5/site-packages/zhihu/topic.py", line 269, in questions
    res = self._session.get(question_url, params=params)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 480, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/adapters.py", line 426, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))

@Lukasa
Copy link
Member

Lukasa commented Jan 12, 2016

@laike9m Please read the issue. As explained above, you are misreading the traceback. The actual exception you're hitting is the last one in the tree. That error is caused by the remote server closing the connection.

@laike9m
Copy link

laike9m commented Jan 12, 2016

@Lukasa Alright, understand.

@ghost
Copy link

ghost commented Mar 16, 2017

I still have this issue with request 2.13, python 3.5 as soon as the request timeout. Should the issue be reopened?

@Lukasa
Copy link
Member

Lukasa commented Mar 16, 2017

@Cedric-Venet No you don't. You are seeing the effect of Python 3's exception chaining. Please read this issue very carefully.

In fact, I'm now going to lock this issue because it is an absolute magnet for people chiming in with "me too" comments without reading the issue. For anyone who wants to leap in, here is the TL;DR.

This is a problem to do with how exception tracebacks are being reported in Python 3. PEP 3134 introduced 'chaining exceptions' reporting, which means that when an exception bubbles up, all exceptions that were hit on the way to that original exception are shown. The purpose of this error reporting is to highlight that some exceptions occur in except blocks, and to work out what chain of exceptions was hit. This is potentially very useful: for instance, you can hit an exception after destroying a resource and then attempt to use that resource in the except block, which hits another exception. It's helpful to be able to see both exceptions at once.

The key is that the TypeError raised as the first exception is unrelated to the subsequent ones. In fact, that's the standard control flow in urllib3. This means that this error is never your actual problem. In general, when using Python 3, you should look at the last exception in the chain first and treat that as the actual error to solve.

@psf psf locked and limited conversation to collaborators Mar 16, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests