test_2_KeyboardInterrupt failing on Python 3.5

Originally reported by: Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco)

---

======================================================================
ERROR: test_2_KeyboardInterrupt (cherrypy.test.test_states.ServerStateTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jaraco/Dropbox/code/public/cherrypy/cherrypy/test/test_states.py", line 209, in test_2_KeyboardInterrupt
    self.getPage("/ctrlc")
  File "/Users/jaraco/Dropbox/code/public/cherrypy/cherrypy/test/helper.py", line 349, in getPage
    protocol)
  File "/Users/jaraco/Dropbox/code/public/cherrypy/cherrypy/test/webtest.py", line 255, in getPage
    self.HTTP_CONN, protocol or self.PROTOCOL)
  File "/Users/jaraco/Dropbox/code/public/cherrypy/cherrypy/test/webtest.py", line 558, in openURL
    conn.endheaders()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1079, in endheaders
    self._send_output(message_body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 911, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 854, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 826, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 711, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 702, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
-------------------- >> begin captured logging << --------------------
cherrypy.error: INFO: [06/Feb/2016:08:17:31] ENGINE Bus STARTING
cherrypy.error: INFO: [06/Feb/2016:08:17:31] ENGINE Started monitor thread 'Session cleanup'.
cherrypy.error: INFO: [06/Feb/2016:08:17:31] ENGINE Started monitor thread '_TimeoutMonitor'.
cherrypy.error: INFO: [06/Feb/2016:08:17:32] ENGINE Serving on http://127.0.0.1:54583
cherrypy.error: INFO: [06/Feb/2016:08:17:32] ENGINE Bus STARTED
cherrypy.error: INFO: [06/Feb/2016:08:17:32] ENGINE Already serving on http://127.0.0.1:54583
cherrypy.access.4400260936: INFO: 127.0.0.1 - - [06/Feb/2016:08:17:32] "GET / HTTP/1.1" 200 11 "" ""
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE <Ctrl-C> hit: shutting down HTTP server
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Bus STOPPING
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 54583)) shut down
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Stopped thread '_TimeoutMonitor'.
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Stopped thread 'Session cleanup'.
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Bus STOPPED
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Bus EXITING
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Bus EXITED
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Waiting for child threads to terminate...
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Bus STARTING
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Started monitor thread 'Session cleanup'.
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Started monitor thread '_TimeoutMonitor'.
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Serving on http://127.0.0.1:54583
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Bus STARTED
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Already serving on http://127.0.0.1:54583
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE <Ctrl-C> hit: shutting down HTTP server
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Bus STOPPING
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 54583)) shut down
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Stopped thread '_TimeoutMonitor'.
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Stopped thread 'Session cleanup'.
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Bus STOPPED
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Bus EXITING
cherrypy.error: INFO: [06/Feb/2016:08:17:33] ENGINE Bus EXITED
--------------------- >> end captured logging << ---------------------

---

• Bitbucket: https://bitbucket.org/cherrypy/cherrypy/issue/1406

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

The error seems to be related to how connections are retried now. I'm not sure why a call to endheaders should trigger a reconnect. In what world is it viable to reconnect a dropped HTTP connection and resume at the endheaders stage? It seems a little naive to attempt a reconnect during any send call.

I'm tempted in the test to just be more lenient and catch that error as well.

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

I tried replicating the failure using a simple http server that disconnects on request:

import http.server


class DisconnectingHandler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        self.connection.close()


def run():
    httpd = http.server.HTTPServer(("", 8000), DisconnectingHandler)
    httpd.handle_request()


if __name__ == '__main__':
    run()

And calling it simply with the client:

import http.client
conn = http.client.HTTPConnection('localhost', 8000)
conn.putrequest('GET', '/', skip_host=True, skip_accept_encoding=True)
conn.putheader('Host', 'localhost')
conn.endheaders()

But that doesn't fail. I'm now starting to think that the monkeypatch of putrequest on the HTTPConnection in webtest may be at fault.

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

I've removed the monkeypatch, but the failure remains the same.

Original comment by Anonymous:

---

I'm packaging CherryPy into Fedora and I've encountered the very same test failure when building against python 3.5 (for upcoming fedora24).

Here's the buildlog: https://copr-be.cloud.fedoraproject.org/results/torsava/python3-cherrypy/fedora-rawhide-x86_64/00163068-python3-cherrypy/build.log.gz

Note that this is after I've already applied a patch made from pull request #120 that successfully solves issue #1382.

Original comment by Joel Rivera (Bitbucket: cyraxjoe, GitHub: cyraxjoe):

---

The issue caused because of a new exception on the http.client module, the RemoteDisconnected exception. Is specially problematic because it inherits from socket.error which since 3.3 is an alias to OSError.

I have a fix for this issue on my fork. Which I'll probably integrate the change to cherrypy/default on the weekend if no concerns or other alternatives are suggested.

Original comment by Anonymous:

---

@cyraxjoe With the patch from your fork, CherryPy now successfully builds against Python 3.5 in Fedora 24, thanks for your work!

Original comment by Joel Rivera (Bitbucket: cyraxjoe, GitHub: cyraxjoe):

---

Add new argument to openURL for the testsuite to be able to
raise subclasses of socket.error (since python3.3 OSError)
instead of retrying.

This fixes the KeyboardError problem for 3.5 by specifying that
subclasses of BadStatusLine can be raised.

The longer explanation of the fix:

From python3.5 a new exception is retuned when the connection
ends abruptly:
  http.client.RemoteDisconnected
RemoteDisconnected is a subclass of:
  (ConnectionResetError, http.client.BadStatusLine)
and ConnectionResetError is an indirect subclass of:
   OSError
From python 3.3 an up socket.error is an alias to OSError
following PEP-3151, therefore http.client.RemoteDisconnected
is considered a socket.error.

raise_subcls specifies the classes that are not going
to be considered as a socket.error for the retries.
Given that RemoteDisconnected is part BadStatusLine
we can use the same call for all py3 versions without
sideffects. python < 3.5 will raise directly BadStatusLine
which is not a subclass for socket.error/OSError.

Fixes issue #1406.