Tendril Frame-based Network Connection Tracker
Tendril is a network communication library based on two main features:
it is based on sending and receiving frames, and it tracks the state
of an abstract connection as defined by the application. Tendril is
designed to be easy to use: creating an application requires
Application class and providing an implementation
for the recv_frame() method; then get a
instance and start it, and Tendril manages the rest.
Tendril is based on the concept of passing around frames or packets of data. The fact is, most network protocols are based on sending and receiving frames; for instance, in the SMTP protocol used for sending email, the sender will start off sending the frame "MAIL FROM email@example.com" followed by a line termination sequence (a carriage return followed by a newline). The SMTP server will then respond with another frame acknowledging the "MAIL FROM" frame, and that frame will also end with a line termination sequence. Thus, even though SMTP is defined on top of the TCP protocol, which provides an undivided stream of data between the client and server, a framing boundary is imposed upon it--in this case, the carriage return followed by a newline that terminates each frame.
Tendril includes the concept of framers. A framer is nothing more
than a subclass of
Framer which has one method which extracts a
single frame from the stream of undifferentiated data, and another
method which converts a frame into an appropriate representation. In
the case of the SMTP protocol exchange above, the
method finds each line terminated by the carriage return-newline pair,
strips off those characters, and returns just the frame. In the same
way, the corresponding
streamify() method takes the frame and
appends a carriage return-newline pair.
For text-based protocols such as SMTP, this may seem like overkill. However, for binary-based protocols, a lot of code is dedicated to determining the boundaries between frames, and in some cases even decoding the frame. Tendril's concept of a framer for a connection enables the framing logic to be isolated from the rest of the application, and even reused: Tendril comes with several pre-built framers, including framers designed to work with a text-based protocol such as SMTP.
Another important advantage of the framer concept is the ability to switch between framers as needed. Taking again the example of the SMTP protocol--the actual email data is transferred to the server by the client first sending a "DATA" frame; the server responds indicating that it is ready to begin receiving the message data, and then the client simply sends the message data, ending it with a line containing only a single period ("."). In this case, an SMTP server application based on Tendril may wish to receive the message data as a single frame; it can do this by creating a framer which buffers stream data until it sees that ending sentinel (the period on a line by itself), then returns the whole message as a single frame. Once the server receives the "DATA" frame from the client, all it has to do is temporarily switch out the framer in use for the receiving side of the connection, then switch it back to the standard line-based framer once it has received the message frame.
Tendril allows for different framers to be used on the receiving side and sending side of the connection. This could be used in a case like the SMTP server example cited above, where the server still wishes to send line-oriented frames to the client, even while buffering a message data frame. In addition, although the provided framers deal with byte data, Tendril itself treats the frames as opaque; applications can use this to build a framer that additionally parses a given frame into a class object that the rest of the application then processes as necessary.
Tendril is also based on the concept of tracking connection state. For connection-oriented protocols such as TCP, obviously, this is not a big problem; however, Tendril is also designed to support connectionless protocols such as UDP, where some applications need to manage state information relevant to a given exchange. As an admittedly contrived example, consider DNS, which is based on UDP. A client of the DNS system will send a request to a DNS server over UDP; when a response is received from that DNS server, the connection state information tracked by Tendril can help connect that response with the appropriate request, ensuring that the response goes to the right place.
This connection state tracking is primarily intended to assist applications which desire to be available over both connection-oriented protocols such as TCP and over connectionless protocols such as UDP. Although Tendril does not address reliability or frame ordering, its connection state tracking eases the implementation of an application which utilizes both types of protocols.
Careful readers may have noticed the use of the terms, "such as TCP"
and "such as UDP." Although Tendril only has built-in support for TCP
and UDP connections, it is possible to extend Tendril to support other
protocols. All that is required is to create subclasses of
Tendril (representing an individual connection) and of
TendrilManager (which accepts and creates connections and manages
any necessary socket data flows), and to register the
TendrilManager subclasses as
pkg_resources entry points under
tendril.manager namespace. See the
setup.py for Tendril
for an example of how this may be done.
In addition to allowing Tendril to support protocols other than TCP
and UDP, it is also possible to implement new framers by subclassing
Framer class. (Note: as Tendril deals with
objects, it is not necessary to register these framers using
pkg_resources entry points.) Objects of these classes may then
simply be assigned to the appropriate
framers attribute on the
Tendril instance representing the connection.
Tendril also provides an advanced interface that allows a given raw socket to be "wrapped." Using this feature, an ordinary TCP socket could be converted into an SSL socket. Other uses for this interface are possible, such as setting socket options for the socket. Tendril also provides an interface to allow multiple of these wrapper functions to be called in a given order.
The first step in using Tendril is to define an application by
Application class. (Subclassing is not strictly
necessary--Tendril uses Python's standard
abc package for defining
abstract base classes--but using subclassing will pull in a few
helpful and/or required methods.) The subclass need merely implement
the recv_frame() method, which will be called when a frame is
Application subclass constructor itself can be the
acceptor to be used by Tendril (more on acceptors in a moment).
Application subclass has been created, the developer then
needs to get a
TendrilManager instance, using the
get_manager() factory function. The exact call to
get_manager() depends on the needs; for making outgoing
connections, simply calling
get_manager("tcp") is sufficient. If
listening on a port or making an outgoing connection from a specific
address and/or port is desired, the second argument to
get_manager() may be a tuple of the desired local IP address and
the port number (i.e.,
All managers must be started, and
get_manager() does not start the
manager by itself. Check the manager's
running attribute to see
if the manager is already running, and if it is not, call its
start() method. To accept connections, pass
acceptor (usually the
Application subclass). The
method also accepts a wrapper, which will be called with the
listening socket when it is created.
If, instead of accepting connections (as a server would do), the
desire is to make outgoing connections, simply call
no arguments, then call the
connect() method of the manager. This
method takes the target of the connection (i.e., the IP address and
port number, as a tuple) and the acceptor. (It also has an optional
wrapper, which will be called with the outgoing socket just prior to
initiating the connection.)
An acceptor is simply a callable taking a single argument--the
Tendril instance representing the connection--and returning an
instance of a subclass of
Application, which will be assigned to
application attribute of the
Tendril instance. The
acceptor initializes the application; it also has the opportunity to
Tendril, such as setting framers, calling the
wrap() method, or simply closing the
TendrilManager does not provide the opportunity to
pass arguments to the acceptor, it is certainly possible to do so.
The standard Python
functools.partial() is one obvious interface,
but Tendril additionally provides its own
the advantage of
TendrilPartial is that the positional argument
passed to the acceptor--the
Tendril instance--will be the first
positional argument, rather than the last one, as would be the case
A wrapper is simply a callable again taking a single argument--in
this case, the socket object--and returning a wrapped version of that
argument; that wrapped version of the socket will then be used in
subsequent network calls. A wrapper which manipulates socket options
can simply return the socket object which was passed in, while one
which performs SSL encapsulation can return the SSL wrapper. Again,
although there is no opportunity to pass arguments to the wrapper in a
connect() call (or a
functools.partial() or Tendril's
TendrilPartial utility can be used. In particular, in conjunction
ssl.wrap_socket() call can be used as
a socket wrapper directly, enabling an SSL connection to be set up
Of course, it may be necessary to perform multiple "wrapping"
activities on a connection, such as setting socket options followed by
wrapping the socket in an SSL connection. For this case, Tendril
WrapperChain; it can be initialized in the same way
TendrilPartial is, but additional wrappers can be added by
chain() method; when called, the
object will call each wrapper in the order defined, returning the
final wrapped socket in the end.