int overflow errors on large buffers

[marcinkuzminski]

Org traceback can be found here:
https://bitbucket.org/marcinkuzminski/rhodecode/issue/838

2013-05-13 14:57:05.390 ERROR [waitress] Exception when serving /rhodecode/Machinima-Project
Traceback (most recent call last):
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/waitress/channel.py", line 329, in service
    task.service()
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/waitress/task.py", line 173, in service
    self.execute()
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/waitress/task.py", line 412, in execute
    self.write(chunk)
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/waitress/task.py", line 303, in write
    channel.write_soon(towrite)
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/waitress/channel.py", line 311, in write_soon
    self.outbufs[-1].append(data)
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/waitress/buffers.py", line 247, in append
    sz = len(buf)
OverflowError: long int too large to convert to int
Traceback (most recent call last):
  File "/usr/share/rhodecode/rhodecode-venv/bin/paster", line 8, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/paste/script/serve.py", line 311, in command
    serve()
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/paste/script/serve.py", line 295, in serve
    server(app)
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 189, in server_wrapper
    **context.local_conf)
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/paste/deploy/util.py", line 56, in fix_call
    val = callable(*args, **kw)
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/waitress/__init__.py", line 21, in serve_paste
    serve(app, **kw)
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/waitress/__init__.py", line 18, in serve
    server.run()
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/waitress/server.py", line 129, in run
    self.asyncore.loop(map=self._map)
  File "/usr/lib/python2.6/asyncore.py", line 210, in loop
    poll_fun(timeout, map)
  File "/usr/lib/python2.6/asyncore.py", line 127, in poll
    is_r = obj.readable()
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/waitress/channel.py", line 157, in readable
    self.any_outbuf_has_data())
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/waitress/channel.py", line 84, in any_outbuf_has_data
    if bool(outbuf):
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/waitress/buffers.py", line 214, in __nonzero__
    return bool(len(self))
  File "/usr/share/rhodecode/rhodecode-venv/lib/python2.6/site-packages/waitress/buffers.py", line 209, in __len__
    return len(buf)
OverflowError: long int too large to convert to int
Removing PID file /usr/share/rhodecode/rhodecode.pid

So basically python expects an int when it calls len(), it seems that buffer length can exceed that , and len functions on buffers can return more than that.

Seems to be hard to fix issue, not sure if that should be fixed even ?

[tseaver]

I don't know that anybody contemplated using a single buffer for more than 2Gb of data: seems a bit extreme.

We could code around the particular crash by avoiding the call to len(self.buf), e.g.:

def __nonzero(self):
    return self.__len__() > 0

[marcinkuzminski]

2GB is not much when it comes to pushing large repos, (this is what user was doing i assume).

Your fix wouldn't help i think because it crashes inside __len__ when len(self.buf) is called.

I wrote small test and a fix that looks like this:

    def __len__(self):
        buf = self.buf
        if buf is not None:
            return buf.__len__()
        else:
            return self.strbuf.__len__()

    def __nonzero__(self):
        return self.__len__() > 0

I can open a PR if that would be accepted

[marcinkuzminski]

I found another case of this:

Traceback (most recent call last): 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/channel.py", line 332, in service 
    task.service() 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/task.py", line 173, in service 
    self.execute() 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/task.py", line 420, in execute 
    self.write(chunk) 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/task.py", line 303, in write 
    channel.write_soon(towrite) 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/channel.py", line 314, in write_soon 
    self.outbufs[-1].append(data) 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/buffers.py", line 244, in append 
    sz = len(buf) 
OverflowError: long int too large to convert to int 
2013-11-14 17:24:28.508 ERROR [waitress] Unexpected exception when flushing 
Traceback (most recent call last): 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/channel.py", line 137, in handle_write 
    flush() 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/channel.py", line 236, in _flush_some 
    outbuflen = len(outbuf) 
OverflowError: long int too large to convert to int

[tseaver]

Is this with the current release (0.8.7)? Because 'outbuf' should be an instance of the OverflowableBuffer whose 'len' you helped fix earlier.

Hmm, it does look like there is one case where a non-OB is added to 'self.outbufs':

https://github.com/Pylons/waitress/blob/master/waitress/channel.py#L310

So, I guess we need to fix the ReadOnlyFileBasedBuffer too. But this should really be a new issue (or better, PR).

[marcinkuzminski]

I created new issue #47