Another int overflow on waitress Buffers #47

Closed
marcinkuzminski opened this Issue Nov 15, 2013 · 9 comments

Projects

None yet

2 participants

@marcinkuzminski
Member

So it's another case of #22, now on waitress 0.8.7

Org exception was:

Traceback (most recent call last): 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/channel.py", line 332, in service 
    task.service() 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/task.py", line 173, in service 
    self.execute() 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/task.py", line 420, in execute 
    self.write(chunk) 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/task.py", line 303, in write 
    channel.write_soon(towrite) 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/channel.py", line 314, in write_soon 
    self.outbufs[-1].append(data) 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/buffers.py", line 244, in append 
    sz = len(buf) 
OverflowError: long int too large to convert to int 
2013-11-14 17:24:28.508 ERROR [waitress] Unexpected exception when flushing 
Traceback (most recent call last): 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/channel.py", line 137, in handle_write 
    flush() 
  File "/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/channel.py", line 236, in _flush_some 
    outbuflen = len(outbuf) 
OverflowError: long int too large to convert to int

I did two changes just to test out:

-    outbuflen = len(outbuf)
+   outbuflen = outbuf.__len__()

and

- sz = len(buf)
+ sz = buf.__len__()

Then we end up with:

2013-11-15 15:05:47.369 ERROR [waitress] uncaptured python exception, 
closing channel <waitress.channel.HTTPChannel connected 127.0.0.1:33700 at 
0xc1a9a0c> (<type 'exceptions.OverflowError'>:long int too large to convert 
to int [/usr/lib/python2.6/asyncore.py|write|86] 
[/usr/lib/python2.6/asyncore.py|handle_write_event|450] 
[/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/channel.py| 
handle_write|124] 
[/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/channel.py| 
total_outbufs_len|92]) 
2013-11-15 15:05:47.369 ERROR [waitress] Unknown exception while trying to 
close outbuf 
Traceback (most recent call last): 
  File 
"/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/channel.py", 
 line 273, in handle_close 
    outbuf._close() 
  File 
"/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/buffers.py", 
 line 295, in _close 
    buf._close() 
  File 
"/data/rhodecodeProduktion/rhodecode-venv/lib/python2.6/site-packages/waitress/buffers.py", 
 line 110, in _close 
    self.file.close() 
IOError: close() called during concurrent operation on the same file 
object.

@marcinkuzminski
Member

I did this test to mimic the issue on append():

    def test_create_buffer_with_remain_more_than_max_int(self):
        from waitress.buffers import OverflowableBuffer, TempfileBasedBuffer

        class IntOverflowBuffer(TempfileBasedBuffer):

            @property
            def remain(self):
                # maxint + 1
                return 0x7fffffffffffffff + 1

            @remain.setter
            def remain(self, data):
                pass

        inst = self._makeOne()
        inst.buf = OverflowableBuffer(False)
        # mimick _set_large_buffer, method
        inst.buf.buf = IntOverflowBuffer()
        inst.overflowed = True

        # append data to already huge buffer
        inst.buf.append(b'x')

But i have no idea how to reliable test the handle_write() and flush() issue.

@marcinkuzminski
Member

@tseaver any idea how to test the handle_write() ?

@mcdonc
Member
mcdonc commented Nov 20, 2013

@marcinkuzminski can you tell whether this error is raised in the situation that @tseaver suggested? He suggested it happens when a ReadOnlyFileBasedBuffer is added to self.outbufs at https://github.com/Pylons/waitress/blob/master/waitress/channel.py#L310 . It'd mean that you were using wsgi.file_wrapper to serve a file, is that true?

@marcinkuzminski
Member

@mcdonc unfortunately i don't have access to this information (it's a machine of one of our clients)

This error doesn't occur when serving files, it's a clone operation of a huge (around 6GB) mercurial repository.
infact when i checked on creating the Mercurial wsgi app i've got:

'wsgi.file_wrapper': <class 'waitress.buffers.ReadOnlyFileBasedBuffer'>,

in th environ

@mcdonc mcdonc added a commit that referenced this issue Nov 21, 2013
@mcdonc mcdonc - Fix some cases where the creation of extremely large output buffers…
… (greater

  than 2GB, suspected to be buffers added via ``wsgi.file_wrapper``) might
  cause an OverflowError on Python 2.  See
  #47.

See #47
34aa289
@mcdonc
Member
mcdonc commented Nov 21, 2013

The presence of 'wsgi.file_wrapper' in the environ doesn't really mean much; anyone who is using waitress will have this in the environ.

Your test demonstrates that the append method of an OverflowableBuffer should probably use __len__() rather than len(). In addition its prune() method should do the same. Also, your first traceback indicates that the _flush_some() method of the HTTPChannel should be using outbuf.__len__ rather than len(outbuf). And finally, your last traceback (the one that seems to emanate from _close()) indicates that the HTTPChannel's total_outbufs_len() method should also use b.__len__() instead of len(b).

I've checked in these fixes but I'm not totally confident that my commit will fix all of the OverflowError cases; hopefully you can test the waitress master and let us know if you see any more of these in production.

@marcinkuzminski
Member

That was my first reaction to do this (fix len calls), but did you seen what error it triggered ?

@mcdonc
Member
mcdonc commented Nov 21, 2013

Yes. See my explanation above wrt "total_outbufs_len". I fixed more than you did.

@marcinkuzminski
Member

Great, thanks for the very detailed explanation and fixes, i'll push to test latest master on troublesome repositories. I'll share some feedback when we get it.

@pcdude2143 pcdude2143 pushed a commit to pcdude2143/dotfiles that referenced this issue Dec 15, 2013
Michael Shepard Squashed '.vim/bundle/YouCompleteMe.vim/third_party/waitress/' change…
…s from 992dd54..1695585

1695585 update documentation version to 0.8.8 to catch up the package version
29f2475 prep for 0.8.8
8959cd9 - When the ``--unix-socket-perms`` option was used as an argument to   ``waitress-serve``, a ``TypeError`` would be raised.  See   Pylons/waitress#50.
e8d1419 - When waitress receives a ``Transfer-Encoding: chunked`` request, we no longer   send the ``TRANSFER_ENCODING`` nor the ``HTTP_TRANSFER_ENCODING`` value to   the application in the environment.  Instead, we pop this header.  Since we   cope with chunked requests by buffering the data in the server, we also know   when a chunked request has ended, and therefore we know the content length.   We set the content-length header in the environment, such that applications   effectively never know the original request was a T-E: chunked request; it   will appear to them as if the request is a non-chunked request with an   accurate content-length.
0110be2 Add a change note, fix test so it passes on Python versions without assertIn, add an NB: to the code.
567d98e Merge branch 'master' of github.com:domruf/waitress into domruf-master
735adb0 - When the ``url_prefix`` adjustment starts with more than one slash, all   slashes except one will be stripped from its beginning.  This differs from   older behavior where more than one leading slash would be preserved in   ``url_prefix``.
fdb9d3e restore py3 compat, coverage
34aa289 - Fix some cases where the creation of extremely large output buffers (greater   than 2GB, suspected to be buffers added via ``wsgi.file_wrapper``) might   cause an OverflowError on Python 2.  See   Pylons/waitress#47.
e9ca506 add test_preserve_header_value_order
b8d453d only sort the headers by key not by value

git-subtree-dir: .vim/bundle/YouCompleteMe.vim/third_party/waitress
git-subtree-split: 169558586d477f6f22402300422b90b5334b3654
a0656c1
@mcdonc
Member
mcdonc commented Jul 14, 2014

@marcinkuzminski I'm closing this issue, as I'm going to assume what I did fixed your production errors. If that's not the case, please reopen.

@mcdonc mcdonc closed this Jul 14, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment