Next evolution of networking code

The poller is basically unchanged, except that it was moved from
neubot/net/ to neubot/.  New stream etc. share most code with
old networing code, except that the interface with high level
protocols, e.g. HTTP, has been radically changed.  (In a certain
sense this is a return to January 2011 networking code, given
that now protocol code has to specify a callback for each slow
I/O operation requested, in order to disentangle the tree.)

The plan is to keep around both the new and the old code, to be
able to refine the new code in tree, and to allow for smooth
migration from the old to the new networking code.  I've been
working in background on that for a long time, but nothing beats
having the code in the tree and experimenting with it.

cert.pem

@@ -0,0 +1,36 @@
+Certificate for 127.0.0.1, for testing OpenSSL support: a number
+of simple/low-level modules assume that this certificate is there
+when you tell them to enable SSL (typically -S option).
+
+-----BEGIN RSA PRIVATE KEY-----
+MIICXQIBAAKBgQCxU/GsN87T+LlGGltclpLE5fy7P9mmkMrjbhPxd3R5/K/VQlKT
+MKt4Ihrr8Zu2DW52z4kodRkMn5U/s4P/SnrlZg89wWvgO0b3Sgf3Oe75ltX+f0o6
+Y1yQJVaWdnIN/btsFooDPaqP5FjTGnDFO2K9MRsQ3PtF2tGdhBrLI2tpxwIDAQAB
+AoGAKyYc+WX/Cu9LzfYd4xK0lfrMm5e202QKKnsfmTMTAZfuBTuFMlG2d5385Qq+
+c/ciuQBZBKIxvTObsotxBA7Qhlcx1sHm/ahXcP1bSfpl7q9voVywPyb2al0Tuixo
+6QhFDYunusbLjkXH531/wtWDHFWurFWMcUlTcqY6ObwfrSECQQDrX4qKXz/WdXMW
+A6Xsnd7Pa7ny9GMzn2sJzySAhURgvvEdLML1byEyfLOFWr2Gnox4rbgYPexdAHmw
+DPCOr3lXAkEAwN4w2fHnslqKf9DD9XWPXgVcWZTaEpv5Tnr73ww+o1NtZdTGr926
+mtY2IW9YOXZ6tnwNGzjAw+jNf+OqrDadEQJBAKdKPLmkgUb4K1gWN7Q5cMeUFZHs
+ySVDxVwvcg42qibpD45g5iBzX/D2WNILcHFh9w0+y33PZVOkptjOGZwQc/kCQCqx
+RGkKFjqxthTC9o2gH0M1tpKR04/o/M+1g4mFIVxv/DhdWDnXwBXEMylFh2b45gL9
+BL2w22LCZrLXh5ElabECQQDA/K4Lw32qPsxXLjrmapT2XuAiLoCddkGkZRCEm4zW
+aJIGvMHMrFUhcl11kGImO1N3fzROnydLQDHvlnJvi2J1
+-----END RSA PRIVATE KEY-----
+-----BEGIN CERTIFICATE-----
+MIICyjCCAjOgAwIBAgIJAPni8Z5efrxNMA0GCSqGSIb3DQEBBQUAMH4xCzAJBgNV
+BAYTAklUMQ4wDAYDVQQIDAVUdXJpbjEOMAwGA1UEBwwFVHVyaW4xFDASBgNVBAoM
+C05leGEgQ2VudGVyMRIwEAYDVQQDDAkxMjcuMC4wLjExJTAjBgkqhkiG9w0BCQEW
+FnNpbW9uZS5iYXNzb0Bwb2xpdG8uaXQwHhcNMTIwOTIyMjIxNTIwWhcNMTMwOTIy
+MjIxNTIwWjB+MQswCQYDVQQGEwJJVDEOMAwGA1UECAwFVHVyaW4xDjAMBgNVBAcM
+BVR1cmluMRQwEgYDVQQKDAtOZXhhIENlbnRlcjESMBAGA1UEAwwJMTI3LjAuMC4x
+MSUwIwYJKoZIhvcNAQkBFhZzaW1vbmUuYmFzc29AcG9saXRvLml0MIGfMA0GCSqG
+SIb3DQEBAQUAA4GNADCBiQKBgQCxU/GsN87T+LlGGltclpLE5fy7P9mmkMrjbhPx
+d3R5/K/VQlKTMKt4Ihrr8Zu2DW52z4kodRkMn5U/s4P/SnrlZg89wWvgO0b3Sgf3
+Oe75ltX+f0o6Y1yQJVaWdnIN/btsFooDPaqP5FjTGnDFO2K9MRsQ3PtF2tGdhBrL
+I2tpxwIDAQABo1AwTjAdBgNVHQ4EFgQUpKn3gGWEH3dNCfE/9A/TKJyNemcwHwYD
+VR0jBBgwFoAUpKn3gGWEH3dNCfE/9A/TKJyNemcwDAYDVR0TBAUwAwEB/zANBgkq
+hkiG9w0BAQUFAAOBgQAIGy+694K2IOJOTZVKIiK1Xc95Jh+i6tXY6GkjkjS0nnYW
+ribgYpjKQGeC/kTm5kY344LKdPoS+oDxWUP3SxT5Q8LKnoanj7HgW6vDZLQPEA0E
+Zqkkkd8Tl/KzL0YG7SENuSy7dTzLZmqPgvqmUBL9HQBUbYpA2+GRDwqnD9+O+g==
+-----END CERTIFICATE-----

doc/neubot/networking_code.txt

@@ -0,0 +1,38 @@
+Neubot networking code
+''''''''''''''''''''''
+
+:Authors: Simone Basso <bassosimone@gmail.com>
+:Version: 1.1
+:Date: 2012/09/27
+:X-Documents: cert.pem neubot/brigade.py neubot/connector.py neubot/handler.py
+    neubot/listener.py neubot/net/poller.py neubot/net/stream.py
+    neubot/pollable.py neubot/poller.py neubot/sslstream.py neubot/stream.py
+
+The networking code is the core of Neubot.  It is based on a global POLLER
+object (implemented in neubot/poller.py), which polls Pollable objects (defined
+in neubot/pollable.py) for readability and writability.  The poller also takes
+care of scheduling and dispatching future events, by using standard library's
+event scheduler (Lib/sched.py).
+
+At the moment of writing this note there are three different registered pollable
+objects: connected stream sockets (neubot/stream.py), listening stream sockets
+(neubot/listener.py), and connect-pending stream sockets (neubot/connector.py).
+The complexity of listening and connect-pending stream sockets is partially
+hidden by the handler (neubot/handler.py), which is an object that can handle
+a set of connected stream sockets.
+
+Extra support modules are: the SSL stream module (neubot/sslstream.py), which
+extends the base stream module to add support for SSL (you typically don't
+need to use this module directly, since neubot/stream.py imports and uses it
+when SSL support is requested); the bucket brigade module (neubot/brigade.py),
+loosely inspired by Apache brigades, which basically simplifies the task of
+bufferising and reading incoming network data.  Worth mentioning is also the
+127.0.0.1-only certificate file (cert.pem), created for the purpose of testing
+the SSL code.
+
+Finally, there are the backward-compatibility poller (neubot/net/poller.py) and
+the backward compatibility stream (neubot/net/stream.py), which basically are
+just the previous evolution of the networking code (the new code shares many
+lines of code with the old one, the difference being mainly in how the new code
+interfaces with protocol objects, e.g. HTTP).  Old files are kept in tree to
+allow for a smooth migration from the old to the new networking code.

neubot/brigade.py

@@ -0,0 +1,91 @@
+# neubot/brigade.py
+
+#
+# Copyright (c) 2012 Simone Basso <bassosimone@gmail.com>,
+#  NEXA Center for Internet & Society at Politecnico di Torino
+#
+# This file is part of Neubot <http://www.neubot.org/>.
+#
+# Neubot is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Neubot is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Neubot.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+''' Bucket brigade '''
+
+from collections import deque
+from neubot import six
+
+NEWLINE = six.b('\n')
+EMPTY = six.b('')
+
+class Brigade(object):
+
+    ''' Bucket brigade '''
+
+    def __init__(self):
+        self.brigade = deque()
+        self.total = 0
+
+    def bufferise(self, octets):
+        ''' Bufferise incoming data '''
+        self.brigade.append(octets)
+        self.total += len(octets)
+
+    def skip(self, length):
+        ''' Skip up to lenght bytes from brigade '''
+        if self.total >= length:
+            while length > 0:
+                bucket = self.brigade.popleft()
+                if len(bucket) > length:
+                    self.brigade.appendleft(six.buff(bucket, length))
+                    self.total -= length
+                    return 0
+                length -= len(bucket)
+                self.total -= len(bucket)
+        return length
+
+    def pullup(self, length):
+        ''' Pullup length bytes from brigade '''
+        retval = []
+        if self.total >= length:
+            while length > 0:
+                bucket = self.brigade.popleft()
+                if len(bucket) > length:
+                    self.brigade.appendleft(six.buff(bucket, length))
+                    bucket = six.buff(bucket, 0, length)
+                retval.append(str(bucket))
+                self.total -= len(bucket)
+                length -= len(bucket)
+        return EMPTY.join(retval)
+
+    def getline(self, maxline):
+        ''' Read line from brigade '''
+        if self.total >= maxline:
+            tmp = self.pullup(maxline)
+        else:
+            tmp = self.pullup(self.total)
+            self.brigade.clear()
+            self.total = 0
+        index = tmp.find(NEWLINE)
+        if index >= 0:
+            line = tmp[:index + 1]
+            remainder = tmp[index + 1:]
+            if remainder:
+                self.brigade.appendleft(remainder)
+                self.total += len(remainder)
+            return line
+        if len(tmp) >= maxline:
+            raise RuntimeError('brigade: line too long')
+        self.brigade.appendleft(tmp)
+        self.total += len(tmp)
+        return EMPTY

neubot/connector.py

@@ -0,0 +1,102 @@
+# neubot/connector.py
+
+#
+# Copyright (c) 2010-2012 Simone Basso <bassosimone@gmail.com>,
+#  NEXA Center for Internet & Society at Politecnico di Torino
+#
+# This file is part of Neubot <http://www.neubot.org/>.
+#
+# Neubot is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Neubot is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Neubot.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+''' Pollable socket connector '''
+
+# Adapted from neubot/net/stream.py
+
+import collections
+
+from neubot.pollable import Pollable
+from neubot.poller import POLLER
+
+from neubot import utils_net
+from neubot import utils
+
+class Connector(Pollable):
+
+    ''' Pollable socket connector '''
+
+    def __init__(self, parent, endpoint, prefer_ipv6, sslconfig, extra):
+        Pollable.__init__(self)
+
+        self.epnts = collections.deque()
+        self.parent = parent
+        self.prefer_ipv6 = prefer_ipv6
+        self.sslconfig = sslconfig
+        self.extra = extra
+        self.sock = None
+        self.timestamp = 0
+        self.watchdog = 10
+
+        # For logging purpose, save original endpoint
+        self.endpoint = endpoint
+
+        if " " in endpoint[0]:
+            for address in endpoint[0].split():
+                tmp = (address.strip(), endpoint[1])
+                self.epnts.append(tmp)
+        else:
+            self.epnts.append(endpoint)
+
+        self._connect()
+
+    def __repr__(self):
+        return str(self.endpoint)
+
+    def _connection_failed(self):
+        ''' Failed to connect first available epnt '''
+        if self.sock:
+            POLLER.unset_writable(self)
+            self.sock = None  # MUST be below unset_writable()
+        self.epnts.popleft()
+        if not self.epnts:
+            self.parent.handle_connect_error(self)
+            return
+        self._connect()
+
+    def _connect(self):
+        ''' Connect first available epnt '''
+        sock = utils_net.connect(self.epnts[0], self.prefer_ipv6)
+        if sock:
+            self.sock = sock
+            self.timestamp = utils.ticks()
+            POLLER.set_writable(self)
+        else:
+            self._connection_failed()
+
+    def fileno(self):
+        return self.sock.fileno()
+
+    def handle_write(self):
+        POLLER.unset_writable(self)
+
+        if not utils_net.isconnected(self.endpoint, self.sock):
+            self._connection_failed()
+            return
+
+        self.parent.handle_connect(self, self.sock,
+               (utils.ticks() - self.timestamp),
+               self.sslconfig, self.extra)
+
+    def handle_close(self):
+        self._connection_failed()

neubot/defer.py

@@ -45,6 +45,9 @@ class Deferred(object):
     def __init__(self):
         self.chain = collections.deque()
 
+    def __len__(self):
+        return len(self.chain)
+
     def add_callback(self, func):
         ''' Add a callback to the deferred '''
         self.chain.append((CALLBACK, func))

neubot/handler.py

@@ -0,0 +1,73 @@
+# neubot/handler.py
+
+#
+# Copyright (c) 2010-2012 Simone Basso <bassosimone@gmail.com>,
+#  NEXA Center for Internet & Society at Politecnico di Torino
+#
+# This file is part of Neubot <http://www.neubot.org/>.
+#
+# Neubot is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Neubot is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Neubot.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+''' Handles poller events '''
+
+# Adapted from neubot/net/stream.py
+
+from neubot.connector import Connector
+from neubot.listener import Listener
+
+from neubot import utils_net
+
+class Handler(object):
+
+    ''' Event handler '''
+
+    # Inspired by BitTorrent handle class
+
+    def listen(self, endpoint, prefer_ipv6, sslconfig, sslcert):
+        ''' Listen() at endpoint '''
+        sockets = utils_net.listen(endpoint, prefer_ipv6)
+        if not sockets:
+            self.handle_listen_error(endpoint)
+            return
+        for sock in sockets:
+            Listener(self, sock, endpoint, sslconfig, sslcert)
+
+    def handle_listen_error(self, endpoint):
+        ''' Handle the LISTEN_ERROR event '''
+
+    def handle_listen(self, listener):
+        ''' Handle the LISTEN event '''
+
+    def handle_listen_close(self, listener):
+        ''' Handle the LISTEN_CLOSE event '''
+
+    def handle_accept(self, listener, sock, sslconfig, sslcert):
+        ''' Handle the ACCEPT event '''
+
+    def handle_accept_error(self, listener):
+        ''' Handle the ACCEPT_ERROR event '''
+
+    def connect(self, endpoint, prefer_ipv6, sslconfig, extra):
+        ''' Connect() to endpoint '''
+        Connector(self, endpoint, prefer_ipv6, sslconfig, extra)
+
+    def handle_connect_error(self, connector):
+        ''' Handle the CONNECT_ERROR event '''
+
+    def handle_connect(self, connector, sock, rtt, sslconfig, extra):
+        ''' Handle the CONNECT event '''
+
+    def handle_close(self, stream):
+        ''' Handle the CLOSE event '''