BufferError: Existing exports of data: object cannot be re-sized

[rahultaing]

We have a single threaded application using tornado 4.5.3. The application calls stream.write to flush data downstream. We wrote a UT in which we mock the write_to_fd to return 0 bytes. We are trying to replicate a scenario in which the self._write_buffer would continually grow if no bound has been specified.

This test results in a BufferError every time the self._write_buffer is being updated either in write method self._write_buffer += data or in _handle_write method del self._write_buffer[:self._write_buffer_pos]

Let me know if there is other information we can share to help root cause this issue.

[bdarnell]

Can you post this unit test? Does it reproduce with newer versions of Tornado?

[rahultaing]

Here's what the test looks like. AsyncStatsite is our statsite client that is using tornado tcpclient to connect to statsrelay and flush data periodically or when the buffer gets full.

@pytest.mark.gen_test
def test_stream_write_buffer_error(io_loop):
    statsite = AsyncStatsite("127.0.0.1", 8125, io_loop)

    def log_statsite():
        with patch.object(tornado.iostream.IOStream, 'write_to_fd', return_value=0):
            statsite.log(["123" for x in xrange(MESSAGES_TO_BUFFER+1)])

    def stop_ioloop():
        tornado.ioloop.IOLoop.current().stop()

    PeriodicCallback(log_statsite, 1, io_loop).start()
    io_loop.call_later(15, stop_ioloop)

    io_loop.start()

In the AsyncStatsite __init__ we just connect using TCPClient:

 client = tornado.tcpclient.TCPClient()
 self.connect = functools.partial(client.connect, host, port)
 self.stream = self._ioloop.run_sync(self.connect)

And we flush using the write method:
yield self.stream.write(buffer_data)

It doesn't repro on higher versions because that line of code (del self._write_buffer[:self._write_buffer_pos]) has been updated and buffer implementation isn't the same as in version 4.5.3