411 response results in exception in Session.send

[ben-crowhurst]

Full details can be found over at StackOverflow

[sigmavirus24]

So this is probably a stupid question, but does that server actually work? As you discussed with @mjpieters on the question, this doesn't happen with our typical test server. Is there a server we can actually reproduce this with?

[ben-crowhurst]

The application server works with Httplib2, cURL command line tool and browsers (Safari/Chrome).

[Lukasa]

I can see this happening in a number of cases. What I think we'd want to test is whether we can attempt to read from the socket after it threw an exception on the send. Put another way, if we call HTTPConnection.send() and it throws a socket exception, will HTTPConnection.get_response() work well (i.e. no blocking?).

[sigmavirus24]

The application server works with Httplib2, cURL command line tool and browsers (Safari/Chrome).

I understand that it doesn't work with requests. Could you share (even privately, since our emails are publicly accessible) what server it is that you're experiencing problems with so we can attempt to debug this further?

[ben-crowhurst]

Sorry, I'm afraid the client won't allow that to happen :(. I can inform you that placing a sleep(1) after asio::write( response ) within the server code allows Requests to do its job accurately.

Server

1. receive request
2. parse http headers
3. search for content-length
   3.1 if not found return 411
4. parse http body

[Lukasa]

So let's be clear, the problem is that the socket.send() throws an exception because the socket is no longer open for writing. What matters is whether we can still coerce httplib into providing us the response in that case without blocking (in case no such response was made).

[sigmavirus24]

With that detail, we can definitely write a socket-level test. Thank you @ben-crowhurst

[Lukasa]

@sigmavirus24 FYI, if you're planning to take this, I think the fix will actually go into urllib3 (or maybe even httplib, if it turns out httplib can't handle it).

[sigmavirus24]

Yeah. "socket-level test" was my hint at you that this would be in urllib3 ;)

[Lukasa]

So some quick local tests suggest that for EPIPE we're ok. Running this 'server':

import socket

s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('', 8080))
s.listen(0)

while True:
    c = s.accept()[0]
    c.send('HTTP/1.1 411 Length Needed\r\nContent-Length: 0\r\n\r\n')
    c.close()

We can achieve the following in httplib:

>>> import httplib
>>> c = httplib.HTTPConnection('localhost', 8080)
>>> c.putrequest('GET', '/'); c.endheaders(); c.send('some data');
>>> c.send('some data')
>>> c.send('some data')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 826, in send
    self.sock.sendall(data)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 32] Broken pipe
>>> r = c.getresponse()
>>> r.status
411
>>> r.read()
''

So an EPIPE has the potential to give us a response when there is one.

If there isn't a response, it seems that instead we get a BadStatusLine error, which is exactly what I'd suspect. So I think we can bully httplib into doing what we need here.

[ben-crowhurst]

Having looked at the HTTP Adapter I'm suggesting the following alteration?

        except (ProtocolError, socket.error) as err:
            raise ConnectionError(err, request=request, response=response)

This would stop making a behavioural change to the API but allow those interested to respond accordingly.

[Lukasa]

I don't think HTTP Adapter is low enough down. I think urllib3 needs to know what's going on here.

[sigmavirus24]

@Lukasa you're correct.

[sethmlarson]

Closing this as it was never a concern for Requests.