Memory leak in gen engine with try/finally block when connections are closed

[tahajahangir]

(Also posted on http://stackoverflow.com/q/14162950/326792)

This code, shows memory leak in tornado's gen module, when connections are closed without reading the response:

import gc
from tornado import web, ioloop, gen

class MainHandler(web.RequestHandler):
    @web.asynchronous
    @gen.engine
    def get(self):
        gc.collect()
        print len(gc.garbage)  # print zombie objects count
        self.a = '*' * 500000000  # ~500MB data
        CHUNK_COUNT = 100
        try:
            for i in xrange(CHUNK_COUNT):
                self.write('*' * 10000)  # write ~10KB of data
                yield gen.Task(self.flush)  # wait for reciever to recieve
            print 'finished'
        finally:
            print 'finally'

application = web.Application([
    (r"/", MainHandler),
    ])

application.listen(8888)
ioloop.IOLoop.instance().start()

and now, run a simple test client, multiple times

#!/usr/bin/python
import urllib
urlopen('http://127.0.0.1:8888/')  # exit without reading response

Now, server output shows, incremental memory usage:

0
WARNING:root:Write error on 8: [Errno 104] Connection reset by peer
1
WARNING:root:Read error on 8: [Errno 104] Connection reset by peer
WARNING:root:error on read
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/iostream.py", line 361, in _handle_read
    if self._read_to_buffer() == 0:
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/iostream.py", line 428, in _read_to_buffer
    chunk = self._read_from_socket()
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/iostream.py", line 409, in _read_from_socket
    chunk = self.socket.recv(self.read_chunk_size)
error: [Errno 104] Connection reset by peer
2
ERROR:root:Uncaught exception GET / (127.0.0.1)
HTTPRequest(protocol='http', host='127.0.0.1:8888', method='GET', uri='/', version='HTTP/1.0', remote_ip='127.0.0.1', body='', headers={'Host': '127.0.0.1:8888', 'User-Agent': 'Python-urllib/1.17'})
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/web.py", line 1021, in _stack_context_handle_exception
    raise_exc_info((type, value, traceback))
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/web.py", line 1139, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/gen.py", line 120, in wrapper
    runner.run()
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/gen.py", line 345, in run
    yielded = self.gen.send(next)
  File "test.py", line 10, in get
    self.a = '*' * 500000000
MemoryError
ERROR:root:500 GET / (127.0.0.1) 3.91ms

If you set CHUNK_COUNT to 1, the 10KB of data can be written to OS connection buffer, and 'finished' and 'finally' texts will be printed to console, and because generator is completed, no memory leak occurs.

But the strange part is that if your remove the try/finally block, the problem disappears!! (even with CHUNK_COUNT set to 100)

Is this a bug on CPython or tornado or ...?!

[tahajahangir]

One possible fix is manually closing generator on connection close, like this:

class MainHandler(web.RequestHandler):
    def on_connection_close(self):
        self.gen.close()

    @web.asynchronous
    @gen.engine
    def get(self):
        self.gen = self.do_get()
        return self.gen

    def do_get(self):
        ...

[bdarnell]

Have you tried this with the current master branch? I recently fixed a memory leak here (in commit bff0740 ). If that's not it I'll take another look.

[tahajahangir]

Tested with tornado 2.4.1 (so with mentioned commit)

[bdarnell]

OK, so there are several things going on here. First, if the connection is closed, the flush callback is never called, so the generator is just left hanging forever waiting for a callback that never comes (you can see that "finally" is never actually printed to the logs). Normally, the generator would be garbage collected, at which point a GeneratorExit exception would be generated, via a magic del method on the generator object itself. However, in this case the generator is part of a reference cycle, and python's garbage collector does not collect cycles if any object in the cycle has a del method (one surprising bit of magic here is that generators only have a del method if they have a try/except/finally block).

This is tricky to fix properly because of the way many tornado methods signal failure through on_connection_close instead of to their own callback. I think it's relatively easy to fix in this case (I think it would be OK if flush callbacks were run when the connection gets closed, although it would be a problem to do the same for read callbacks. I think I could also break up the reference cycle, but that's hard to do in general), but I need to think about whether there's a more general solution.