Remote end closed connection without response on successive Session requests

[rvernica]

I'm issuing successive GET requests to a Django server and I get Remote end closed connection without response on the second request. If I pause for one second it works fine.

Expected Result

>>> s = requests.Session()
>>> s.get('http://localhost:8000/'); time.sleep(1); s.get('http://localhost:8000')
<Response [404]>
<Response [404]>

Works as expected without Session:

>>> requests.get('http://localhost:8000/'); requests.get('http://localhost:8000')
<Response [404]>
<Response [404]>

Actual Result

>>> s.get('http://localhost:8000/'); s.get('http://localhost:8000')
<Response [404]>
Traceback (most recent call last):
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib64/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.6/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "...local/lib/python3.6/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "...local/lib/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "...local/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib64/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.6/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "...local/lib/python3.6/site-packages/requests/sessions.py", line 525, in get
    return self.request('GET', url, **kwargs)
  File "...local/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "...local/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "...local/lib/python3.6/site-packages/requests/adapters.py", line 495, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.3"
  },
  "idna": {
    "version": "2.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.6"
  },
  "platform": {
    "release": "4.17.19-200.fc28.x86_64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "1010008f",
    "version": "17.3.0"
  },
  "requests": {
    "version": "2.19.1"
  },
  "system_ssl": {
    "version": "1010008f"
  },
  "urllib3": {
    "version": "1.23"
  },
  "using_pyopenssl": true
}

It might be related to #4664.

[nateprewitt]

Hi @rvernica it sounds like this is a product of the Django server closing the connection. It sounds like it's either not configured to handle the number of requests you're sending without a pause, or there's some other race condition. Having an actual server between your service and client should be able to handle connection pooling and queuing. Since we can't do anything about this from Requests perspective, I'm going to close this out.

[rvernica]

Hi @nateprewitt, It is it indeed a Django issue, then why does it work without Session?

[nateprewitt]

@rvernica, the Session object establishes a connection pool and attempts to reuse the same connection for requests to the same host. When you use requests.get we’re establishing a new session for each call.

[rvernica]

So, are you saying the Session keeps the connection open and somehow Django closes it?

Then, could this be related to #4664 and so, not a Django problem?

[nateprewitt]

@rvernica without seeing what's the http message contents and probably the TCP traffic on the connection, it'll be hard to say for sure. Either way, the connection is being forcibly closed by Django while being run locally. If it's a problem with the keep-alive reaping, that's a urllib3 issue as noted in #4664, otherwise it's behaviour of Django. There's not a lot we can do in either case, but I would suggest trying to put Django behind an actual server and see if the behaviour persists.

[kingbuzzman]

@rvernica Any update on this? I'm running into this right now. I'm curious how you got around the issue. Did you open any tickets with django?

[rvernica]

@kingbuzzman I reported the issue to Django here but there seem to be some issues with reproducing it. Maybe you can help confirm it.

[jaap3]

I've posted a way to reproduce the error in the Django ticket mentioned by @rvernica, the question remains who's at fault here?

[sigmavirus24]

Seems that @jaap3 found django/django@934acf1 to be the commit that fixes this in master.