Close old buffer when overflowing in OverflowableBuffer #358
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closed
|
memory usage has been stable the past 20 days since I restarted with the fix |
When switching from a BytesIO to a file backed buffer, we were not closing the old buffer after we had copied its contents. With this change when we switch from a small buffer to a larger buffer or vice-versa we will now attempt to close the old buffer.
digitalresistor
force-pushed
the
bugfix/close-buffer
branch
from
e010d53
to
5193408
Compare
digitalresistor
force-pushed
the
bugfix/close-buffer
branch
from
5193408
to
5e99024
Compare
This was referenced Jan 17, 2022
This was referenced Mar 6, 2022
This was referenced Mar 7, 2022
kraj
pushed a commit
to YoeDistro/meta-openembedded
that referenced
this pull request
Mar 7, 2022
Changelog: ========= Python Version Support ---------------------- - Python 3.6 is no longer supported by Waitress - Python 3.10 is fully supported by Waitress Bugfix ------- - "wsgi.file_wrapper" now sets the "seekable", "seek", and "tell" attributes from the underlying file if the underlying file is seekable. This allows WSGI middleware to implement things like range requests for example See Pylons/waitress#359 and Pylons/waitress#363 - In Python 3 "OSError" is no longer subscriptable, this caused failures on Windows attempting to loop to find an socket that would work for use in the trigger. See Pylons/waitress#361 - Fixed an issue whereby "BytesIO" objects were not properly closed, and thereby would not get cleaned up until garbage collection would get around to it. This led to potential for random memory spikes/memory issues, see Pylons/waitress#358 and Pylons/waitress#357 . Features -------- - When the WSGI app starts sending data to the output buffer, we now attempt to send data directly to the socket. This avoids needing to wake up the main thread to start sending data. Allowing faster transmission of the first byte. See Pylons/waitress#364 - Add REQUEST_URI to the WSGI environment. REQUEST_URI is similar to "request_uri" in nginx. It is a string that contains the request path before separating the query string and decoding "%"-escaped characters. Signed-off-by: Wang Mingyu <wangmy@fujitsu.com> Signed-off-by: Khem Raj <raj.khem@gmail.com>
This was referenced Mar 7, 2022
kraj
pushed a commit
to YoeDistro/meta-openembedded
that referenced
this pull request
Mar 8, 2022
Changelog: ========= Python Version Support ---------------------- - Python 3.6 is no longer supported by Waitress - Python 3.10 is fully supported by Waitress Bugfix ------- - "wsgi.file_wrapper" now sets the "seekable", "seek", and "tell" attributes from the underlying file if the underlying file is seekable. This allows WSGI middleware to implement things like range requests for example See Pylons/waitress#359 and Pylons/waitress#363 - In Python 3 "OSError" is no longer subscriptable, this caused failures on Windows attempting to loop to find an socket that would work for use in the trigger. See Pylons/waitress#361 - Fixed an issue whereby "BytesIO" objects were not properly closed, and thereby would not get cleaned up until garbage collection would get around to it. This led to potential for random memory spikes/memory issues, see Pylons/waitress#358 and Pylons/waitress#357 . Features -------- - When the WSGI app starts sending data to the output buffer, we now attempt to send data directly to the socket. This avoids needing to wake up the main thread to start sending data. Allowing faster transmission of the first byte. See Pylons/waitress#364 - Add REQUEST_URI to the WSGI environment. REQUEST_URI is similar to "request_uri" in nginx. It is a string that contains the request path before separating the query string and decoding "%"-escaped characters. Signed-off-by: Wang Mingyu <wangmy@fujitsu.com> Signed-off-by: Khem Raj <raj.khem@gmail.com>
kraj
pushed a commit
to YoeDistro/meta-openembedded
that referenced
this pull request
Mar 8, 2022
Changelog: ========= Python Version Support ---------------------- - Python 3.6 is no longer supported by Waitress - Python 3.10 is fully supported by Waitress Bugfix ------- - "wsgi.file_wrapper" now sets the "seekable", "seek", and "tell" attributes from the underlying file if the underlying file is seekable. This allows WSGI middleware to implement things like range requests for example See Pylons/waitress#359 and Pylons/waitress#363 - In Python 3 "OSError" is no longer subscriptable, this caused failures on Windows attempting to loop to find an socket that would work for use in the trigger. See Pylons/waitress#361 - Fixed an issue whereby "BytesIO" objects were not properly closed, and thereby would not get cleaned up until garbage collection would get around to it. This led to potential for random memory spikes/memory issues, see Pylons/waitress#358 and Pylons/waitress#357 . Features -------- - When the WSGI app starts sending data to the output buffer, we now attempt to send data directly to the socket. This avoids needing to wake up the main thread to start sending data. Allowing faster transmission of the first byte. See Pylons/waitress#364 - Add REQUEST_URI to the WSGI environment. REQUEST_URI is similar to "request_uri" in nginx. It is a string that contains the request path before separating the query string and decoding "%"-escaped characters. Signed-off-by: Wang Mingyu <wangmy@fujitsu.com> Signed-off-by: Khem Raj <raj.khem@gmail.com>
kraj
pushed a commit
to YoeDistro/meta-openembedded
that referenced
this pull request
Mar 9, 2022
Changelog: ========= Python Version Support ---------------------- - Python 3.6 is no longer supported by Waitress - Python 3.10 is fully supported by Waitress Bugfix ------- - "wsgi.file_wrapper" now sets the "seekable", "seek", and "tell" attributes from the underlying file if the underlying file is seekable. This allows WSGI middleware to implement things like range requests for example See Pylons/waitress#359 and Pylons/waitress#363 - In Python 3 "OSError" is no longer subscriptable, this caused failures on Windows attempting to loop to find an socket that would work for use in the trigger. See Pylons/waitress#361 - Fixed an issue whereby "BytesIO" objects were not properly closed, and thereby would not get cleaned up until garbage collection would get around to it. This led to potential for random memory spikes/memory issues, see Pylons/waitress#358 and Pylons/waitress#357 . Features -------- - When the WSGI app starts sending data to the output buffer, we now attempt to send data directly to the socket. This avoids needing to wake up the main thread to start sending data. Allowing faster transmission of the first byte. See Pylons/waitress#364 - Add REQUEST_URI to the WSGI environment. REQUEST_URI is similar to "request_uri" in nginx. It is a string that contains the request path before separating the query string and decoding "%"-escaped characters. Signed-off-by: Wang Mingyu <wangmy@fujitsu.com> Signed-off-by: Khem Raj <raj.khem@gmail.com>
halstead
pushed a commit
to openembedded/meta-openembedded
that referenced
this pull request
Mar 9, 2022
Changelog: ========= Python Version Support ---------------------- - Python 3.6 is no longer supported by Waitress - Python 3.10 is fully supported by Waitress Bugfix ------- - "wsgi.file_wrapper" now sets the "seekable", "seek", and "tell" attributes from the underlying file if the underlying file is seekable. This allows WSGI middleware to implement things like range requests for example See Pylons/waitress#359 and Pylons/waitress#363 - In Python 3 "OSError" is no longer subscriptable, this caused failures on Windows attempting to loop to find an socket that would work for use in the trigger. See Pylons/waitress#361 - Fixed an issue whereby "BytesIO" objects were not properly closed, and thereby would not get cleaned up until garbage collection would get around to it. This led to potential for random memory spikes/memory issues, see Pylons/waitress#358 and Pylons/waitress#357 . Features -------- - When the WSGI app starts sending data to the output buffer, we now attempt to send data directly to the socket. This avoids needing to wake up the main thread to start sending data. Allowing faster transmission of the first byte. See Pylons/waitress#364 - Add REQUEST_URI to the WSGI environment. REQUEST_URI is similar to "request_uri" in nginx. It is a string that contains the request path before separating the query string and decoding "%"-escaped characters. Signed-off-by: Wang Mingyu <wangmy@fujitsu.com> Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this pull request
Jan 28, 2024
2.1.2 ----- Bugfix ~~~~~~ - When expose_tracebacks is enabled waitress would fail to properly encode unicode thereby causing another error during error handling. See Pylons/waitress#378 - Header length checking had a calculation that was done incorrectly when the data was received across multple socket reads. This calculation has been corrected, and no longer will Waitress send back a 413 Request Entity Too Large. See Pylons/waitress#376 Security Bugfix ~~~~~~~~~~~~~~~ - in 2.1.0 a new feature was introduced that allowed the WSGI thread to start sending data to the socket. However this introduced a race condition whereby a socket may be closed in the sending thread while the main thread is about to call select() therey causing the entire application to be taken down. Waitress will no longer close the socket in the WSGI thread, instead waking up the main thread to cleanup. See Pylons/waitress#377 2.1.1 ----- Security Bugfix ~~~~~~~~~~~~~~~ - Waitress now validates that chunked encoding extensions are valid, and don't contain invalid characters that are not allowed. They are still skipped/not processed, but if they contain invalid data we no longer continue in and return a 400 Bad Request. This stops potential HTTP desync/HTTP request smuggling. Thanks to Zhang Zeyu for reporting this issue. See GHSA-4f7p-27jc-3c36 - Waitress now validates that the chunk length is only valid hex digits when parsing chunked encoding, and values such as ``0x01`` and ``+01`` are no longer supported. This stops potential HTTP desync/HTTP request smuggling. Thanks to Zhang Zeyu for reporting this issue. See GHSA-4f7p-27jc-3c36 - Waitress now validates that the Content-Length sent by a remote contains only digits in accordance with RFC7230 and will return a 400 Bad Request when the Content-Length header contains invalid data, such as ``+10`` which would previously get parsed as ``10`` and accepted. This stops potential HTTP desync/HTTP request smuggling Thanks to Zhang Zeyu for reporting this issue. See GHSA-4f7p-27jc-3c36 2.1.0 ----- Python Version Support ~~~~~~~~~~~~~~~~~~~~~~ - Python 3.6 is no longer supported by Waitress - Python 3.10 is fully supported by Waitress Bugfix ~~~~~~ - ``wsgi.file_wrapper`` now sets the ``seekable``, ``seek``, and ``tell`` attributes from the underlying file if the underlying file is seekable. This allows WSGI middleware to implement things like range requests for example See Pylons/waitress#359 and Pylons/waitress#363 - In Python 3 ``OSError`` is no longer subscriptable, this caused failures on Windows attempting to loop to find an socket that would work for use in the trigger. See Pylons/waitress#361 - Fixed an issue whereby ``BytesIO`` objects were not properly closed, and thereby would not get cleaned up until garbage collection would get around to it. This led to potential for random memory spikes/memory issues, see Pylons/waitress#358 and Pylons/waitress#357 . With thanks to Florian Schulze for testing/vaidating this fix! Features ~~~~~~~~ - When the WSGI app starts sending data to the output buffer, we now attempt to send data directly to the socket. This avoids needing to wake up the main thread to start sending data. Allowing faster transmission of the first byte. See Pylons/waitress#364 With thanks to Michael Merickel for being a great rubber ducky! - Add REQUEST_URI to the WSGI environment. REQUEST_URI is similar to ``request_uri`` in nginx. It is a string that contains the request path before separating the query string and decoding ``%``-escaped characters.
daregit
pushed a commit
to daregit/yocto-combined
that referenced
this pull request
May 22, 2024
Changelog: ========= Python Version Support ---------------------- - Python 3.6 is no longer supported by Waitress - Python 3.10 is fully supported by Waitress Bugfix ------- - "wsgi.file_wrapper" now sets the "seekable", "seek", and "tell" attributes from the underlying file if the underlying file is seekable. This allows WSGI middleware to implement things like range requests for example See Pylons/waitress#359 and Pylons/waitress#363 - In Python 3 "OSError" is no longer subscriptable, this caused failures on Windows attempting to loop to find an socket that would work for use in the trigger. See Pylons/waitress#361 - Fixed an issue whereby "BytesIO" objects were not properly closed, and thereby would not get cleaned up until garbage collection would get around to it. This led to potential for random memory spikes/memory issues, see Pylons/waitress#358 and Pylons/waitress#357 . Features -------- - When the WSGI app starts sending data to the output buffer, we now attempt to send data directly to the socket. This avoids needing to wake up the main thread to start sending data. Allowing faster transmission of the first byte. See Pylons/waitress#364 - Add REQUEST_URI to the WSGI environment. REQUEST_URI is similar to "request_uri" in nginx. It is a string that contains the request path before separating the query string and decoding "%"-escaped characters. Signed-off-by: Wang Mingyu <wangmy@fujitsu.com> Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
daregit
pushed a commit
to daregit/yocto-combined
that referenced
this pull request
May 22, 2024
Changelog: ========= Python Version Support ---------------------- - Python 3.6 is no longer supported by Waitress - Python 3.10 is fully supported by Waitress Bugfix ------- - "wsgi.file_wrapper" now sets the "seekable", "seek", and "tell" attributes from the underlying file if the underlying file is seekable. This allows WSGI middleware to implement things like range requests for example See Pylons/waitress#359 and Pylons/waitress#363 - In Python 3 "OSError" is no longer subscriptable, this caused failures on Windows attempting to loop to find an socket that would work for use in the trigger. See Pylons/waitress#361 - Fixed an issue whereby "BytesIO" objects were not properly closed, and thereby would not get cleaned up until garbage collection would get around to it. This led to potential for random memory spikes/memory issues, see Pylons/waitress#358 and Pylons/waitress#357 . Features -------- - When the WSGI app starts sending data to the output buffer, we now attempt to send data directly to the socket. This avoids needing to wake up the main thread to start sending data. Allowing faster transmission of the first byte. See Pylons/waitress#364 - Add REQUEST_URI to the WSGI environment. REQUEST_URI is similar to "request_uri" in nginx. It is a string that contains the request path before separating the query string and decoding "%"-escaped characters. Signed-off-by: Wang Mingyu <wangmy@fujitsu.com> Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When switching from a BytesIO to a file backed buffer, we were not
closing the old buffer after we had copied its contents. With this
change when we switch from a small buffer to a larger buffer or
vice-versa we will now attempt to close the old buffer.
Closes: #357