cyclone

Info

See github for the latest source.

Author

Alexandre Fiori <fiorix@gmail.com>

About

cyclone is a low-level network toolkit, which provides support for HTTP 1.1 in an API very similar to the one implemented by the Tornado web server.

Key differences between cyclone and tornado

• cyclone is a Twisted protocol, therefore it may be used in conjunction with any other protocol implemented in Twisted.
• Localization is based on the standard Gettext instead of the CSV implementation in the original tornado. The gettext support has been merged back into Tornado.
• Asynchronous HTTP client based on TwistedWeb. It's not compatible with with one provided by Tornado - which is based on PyCurl. (The HTTP server code is NOT based on TwistedWeb, for several reasons)
• Support for sending e-mails based on TwistedMail, with authentication and TLS, plus an easy way to create plain text or HTML messages, and attachments. (see the e-mail demo)
• Support for HTTP Authentication. (see the authentication demo)
• Support for Bottle-like API, Twistd application and Twistd plugin. (see the hello world demos)
• Built-in support for XMLRPC and JsonRPC (rpc demo), Server Sent Events (sse demo) and WebSocket (websocket demo).
• Built-in support for Redis, based on txredisapi. We usually need an in-memory caching server like memcache for web applications. However, we prefer redis over memcache because it supports more operations like pubsub, various data types like sets, hashes (python dict), and persistent storage. (see the redis demo for details.)
• Built-in support for inline SQLite in an API similar to the twisted.enterprise.adbapi.
• XSRF control per RequestHandler. Basically, adding no_xsrf = True in a RequestHandler cause it ignore the _xsrf cookie. This is useful when exposing handlers to HTTP clients other than browsers in the same server that is serving normal HTML content with the _xsrf cookie and form element.

Advantages of being a Twisted Protocol

• Easy deployment of applications, using twistd.
• RDBM (MySQL, PostgreSQL, etc) support via: twisted.enterprise.adbapi.
• NoSQL support for MongoDB (TxMongo) and Redis (TxRedisAPI).
• May combine many more functionality within the webserver: sending emails, communicating with message brokers, etc...

Benchmarks

Check out the benchmarks page.

Development and Deployment

Twisted Plugin is the recommended way to go for production. Create new projects right away off of the generic application skeleton shipped with cyclone:

$ python -m cyclone.app --help

use: cyclone/app.py [options]
Options:
-h --help              Show this help.
-g --git               Use git's name and email settings, and create a git repo on target
-p --project=NAME      Create new cyclone project.
-m --modname=NAME      Use another name for the module [default: project_name]
-v --version=VERSION   Set project version [default: 0.1]
-s --set-pkg-version   Set version on package name [default: False]
-t --target=PATH       Set path where project is created [default: ./]

Example:

python -m cyclone.app -p foobar
cd foobar
twistd -n foobar

Check out the README.rst in the new project's directory for detailed information. It ships with debian init scripts for single or multiple instances (one per cpu core) to help make deployment as simple as possible.

Tips and Tricks

As a clone, the API implemented in cyclone is almost the same of Tornado. Therefore you may use the Tornado Documentation for stuff like templates and so on.

The snippets below will show some tips and tricks regarding the few differences between the two.

Hello World

#!/usr/bin/env python
# coding: utf-8

import cyclone.web
import sys
from twisted.internet import reactor
from twisted.python import log

class MainHandler(cyclone.web.RequestHandler):
    def get(self):
        self.write("Hello, world")

def main():
    log.startLogging(sys.stdout)
    application = cyclone.web.Application([
        (r"/", MainHandler)
    ])

    reactor.listenTCP(8888, application, interface="127.0.0.1")
    reactor.run()

if __name__ == "__main__":
    main()

Twisted Application and Plugin

The advantage of being a Twisted Application is that you don't need to care about basic daemon features like forking, creating pid files, changing application's user and group permissions, and selecting the proper reactor within the code.

Instead, the application may be run by twistd, as follows:

for testing:
/usr/bin/twistd --nodaemon --python=foobar.tac

for production:
/usr/bin/twistd --pidfile=/var/run/foobar.pid \
                --logfile=/var/log/foobar.log \
                --uid=nobody --gid=nobody \
                --reactor=epoll \
                --python=foobar.tac

Following is the Hello World as a twisted application:

# coding: utf-8
# twisted application: foobar.tac

import cyclone.web
from twisted.application import service, internet

class IndexHandler(cyclone.web.RequestHandler):
    def get(self):
        self.write("hello world")

foobar = cyclone.web.Application([(r"/", IndexHandler)])

application = service.Application("foobar")
internet.TCPServer(8888, foobar(),
    interface="127.0.0.1").setServiceParent(application)

Authenticated and Asynchronous decorators

Tornado provides decorator functions for asynchronous and authenticated methods. Obviously, they're also implemented in cyclone, and yet more powerful when combined with a famous Twisted decorator: defer.inlineCallbacks.

The cyclone.web.authenticated decorator may be combined with defer.inlineCallbacks, however, there's a basic rule to use them together. Considering that the authenticated decorator will check user credentials, and, depending on the result, it will continue processing the request OR redirect the request to the login page, it has to be used before the defer.inlineCallbacks to function properly:

class IndexHandler(cyclone.web.RequestHandler):
    @cyclone.web.authenticated
    @defer.inlineCallbacks
    def get(self):
        result = yield something()
        self.write(result)

On the other hand, the cyclone.web.asynchronous decorator will keep the request open until you explicitly call self.finish() later on. Of course, it may also be combined with defer.inlineCallbacks, but it MUST be placed after to function properly:

class Indexhandler(cyclone.web.RequestHandler):
    @defer.inlineCallbacks
    @cyclone.web.asynchronous
    def get(self):
        result = yield something()
        self.finish(result)

Yes, you may combine the three decorators to have the most powerful and simple code in cyclone, like this:

class Indexhandler(cyclone.web.RequestHandler):
    @cyclone.web.authenticated
    @defer.inlineCallbacks
    @cyclone.web.asynchronous
    def get(self):
        try:
            result = yield self.redisdb.get("foo")
        except Exception, e:
            log.msg("Redis query failed: %s" % str(e))
            raise cyclone.web.HTTPError(503) # Service Unavailable

        if not result:
            raise cyclone.web.HTTPError(404)

        self.finish({"result":result})

Localization

The cyclone.locale provides an API based on the Python gettext module.

Because of that, there is one extra option that may be passed to cyclone.locale.load_gettext_translations(path, domain="cyclone"), which the is the gettext's domain. The default domain is cyclone.

Following is a step-by-step guide to implement localization in any cyclone application:

1. Create a python script or twisted application with translatable strings:

   # coding: utf-8
   # twisted application: foobar.tac
   
   import cyclone.web
   import cyclone.locale
   from twisted.application import service, internet
   
   class BaseHandler(cyclone.web.RequestHandler):
       def get_user_locale(self):
           lang = self.get_cookie("lang")
           return cyclone.locale.get(lang)
   
   class IndexHandler(BaseHandler):
       def get(self):
           self.render("index.html")
   
       def post(self):
           _ = self.locale.translate
           name = self.get_argument("name")
           self.write(_("the name is: %s" % name))
   
   class LangHandler(cyclone.web.RequestHandler):
       def get(self, lang):
           if lang in cyclone.locale.get_supported_locales():
               self.set_cookie("lang", lang)
           self.redirect("/")
   
   class Application(cyclone.web.Application):
       def __init__(self):
           handlers = [
               (r"/", IndexHandler),
               (r"/lang/(.+)", LangHandler),
           ]
   
           settings = {
               "static_path": "./static",
               "template_path": "./template",
           }
   
           cyclone.locale.load_gettext_translations("./locale", "foobar")
           cyclone.web.Application.__init__(self, handlers, **settings)
   
   application = service.Application("foobar")
   internet.TCPServer(8888, Application(),
       interface="127.0.0.1").setServiceParent(application)

2. Create a file in ./template/index.html with translatable strings:

   <html>
   <body>
       <form action="/" method="post">
       <p>{{ _("write someone's name:") }}</p>
       <input type="text" name="name">
       <input type="submit" value="{{ _('send') }}">
       </form>
   
       <br>
       <p>{{ _("change language:") }}</p>
       <p><a href="/lang/en_US">English (US)</a></p>
       <p><a href="/lang/pt_BR">Portuguese (BR)</a></p>
   </body>
   </html>

3. Generate PO translatable file from the source code, using xgettext:

   You will notice that xgettext cannot parse HTML properly. It was first designed to parse C files, and now it supports many other languages including Python.

   In order to parse lines like <input type="submit" value="{{ _('send') }}">, you'll need an extra script to pre-process the files.

   Here's what you can use as fix.py:

   #!/usr/bin/env python
   # coding: utf-8
   # fix.py
   
   import re, sys
   
   if __name__ == "__main__":
       try:
           filename = sys.argv[1]
           assert filename != "-"
           fd = open(filename)
       except:
           fd = sys.stdin
   
       line_re = re.compile(r"""['"]{{|}}['"] """)
       for line in fd:
           line = line_re.sub(r"", line)
           sys.stdout.write(line)
       fd.close()

   Then, call xgettext to generate the PO translatable file:

   cat foobar.tac template/index.html | python fix.py | \
       xgettext --language=Python --from-code=utf-8 --keyword=_:1,2 -d foobar

   This will create a file named foobar.po, which needs to be translated, then compiled into an MO file:

   vi foobar.po
   (translate everything, :wq)
   
   mkdir -p ./locale/pt_BR/LC_MESSAGES/
   msgfmt foobar.po -o ./locale/pt_BR/LC_MESSAGES/foobar.mo

4. Finally, test the internationalized application:

   twistd -ny foobar.tac

There is also a complete example with pluralization in demos/locale.

More options and tricks

• Keep-Alive

  Because of the HTTP 1.1 support, sockets aren't always closed when you call self.finish() in a RequestHandler. cyclone lets you enforce that by setting the no_keep_alive attribute attribute in some of your RequestHandlers:

  class IndexHandler(cyclone.web.RequestHandler):
      no_keep_alive = True
      def get(self):
          ...

• Socket closed notification

  One of the great features of TwistedWeb is the request.notifyFinish(), which is also available in cyclone. This method returns a deferred which is fired when the request socket is closed, by either self.finish(), someone closing their browser while receiving data, or closing the connection of a Comet request:

  class IndexHandler(cyclone.web.RequestHandler):
      def get(self):
          ...
          d = self.notifyFinish()
          d.addCallback(remove_from_comet_handlers_list)

• HTTP X-Headers

  When running a cyclone-based application behind Nginx, it's very important to make it automatically use X-Real-Ip and X-Scheme HTTP headers. In order to make cyclone recognize those headers, the option xheaders=True must be set in the Application settings:

  class Application(cyclone.web.Application):
      def __init__(self):
          handlers = [
              (r"/", IndexHandler),
          ]
  
          settings = {
              "xheaders": True
              "static_path": "./static",
          }
  
          cyclone.web.Application.__init__(self, handlers, **settings)

• Cookie-Secret generation

  What I use to generate the "cookie_secrect" key used in cyclone.web.Application's settings is something pretty simple, like this:

  >>> import uuid, base64
  >>> base64.b64encode(uuid.uuid4().bytes + uuid.uuid4().bytes)
  'FoQv5hgLTYCb9aKiBagpJJYtLJInWUcXilg3/vPkUnI='

FAQ

• Where are the request headers?

  They are part of the request, dude:

  class MyHandler(cyclone.web.RequestHandler):
      def get(self):
          # self.request.headers is a dict
          user_agent = self.request.headers.get("User-Agent")

• How do I access raw POST data?

  Both raw POST data and GET/DELETE un-parsed query string are available:

  class MyHandler(cyclone.web.RequestHandler):
      def get(self):
          raw = self.request.query
  
      def post(self):
          raw = self.request.body

• Where is the request information, like remote IP address, HTTP method, URI and version?

  Everything is available as request attributes:

  class MyHandler(cyclone.web.RequestHandler):
      def get(self):
          remote_ip = self.request.remote_ip
          method = self.request.method
          uri = self.request.uri
          version = self.request.version

• How do I set my own headers for the reply?

  Guess what, use self.set_header(name, value):

  class MyHandler(cyclone.web.RequestHandler):
      def get(self):
          self.set_header("Content-Type", "application/json")
          self.finish(cyclone.escape.json_encode({"success":True}))

• What HTTP methods are supported in RequestHandler?

  Well, almost all of them. HEAD, GET, POST, DELETE, PUT and OPTIONS are supported. TRACE is disabled by default, because it may get you in trouble. CONNECT has nothing to do with web servers, it's for proxies.

  For more information on HTTP 1.1 methods, please refer to the RFC 2612 Fielding, et al.. For information regarding TRACE vulnerabilities, please check the following links: What is HTTP TRACE? and Apache Week, security issues.

  Supporting different HTTP methods in the same RequestHandler is easy:

  class MyHandler(cyclone.web.RequestHandler):
      def get(self):
          pass
  
      def head(self):
          pass
  
      ...

Credits

Thanks to (in no particular order):

• Nuswit Telephony API
  • Granting permission for this code to be published and sponsoring
• Gleicon Moraes
  • Testing and using it in the RestMQ web service
• Vanderson Mota
  • Patching setup.py and PyPi maintenance
• Andrew Badr
  • Fixing auth bugs and adding current Tornado's features
• Jon Oberheide
  • Syncing code with Tornado and security features/fixes
• Silas Sewell
  • Syncing code and minor mail fix
• Twitter Bootstrap
  • For making our demo applications look good
• Dan Griffin
  • WebSocket Keep-Alive for OpDemand
• Toby Padilla
  • WebSocket server