Merge pull request #358 from Pylons/bugfix/close-buffer

Close old buffer when overflowing in OverflowableBuffer
Pylons · Jan 17, 2022 · 77276fd · 77276fd
2 parents f41e598 + 5e99024
commit 77276fd
Show file tree

Hide file tree

Showing 5 changed files with 34 additions and 10 deletions.
diff --git a/.github/workflows/ci-tests.yml b/.github/workflows/ci-tests.yml
@@ -15,28 +15,26 @@ jobs:
         strategy:
             matrix:
                 py:
-                    - "3.6"
                     - "3.7"
                     - "3.8"
                     - "3.9"
-                    - "pypy3"
+                    - "3.10"
+                    - "pypy-3.8"
+                    # Pre-release
+                    - "3.11.0-alpha - 3.11.0"
                 os:
                     - "ubuntu-latest"
                     - "windows-latest"
                     - "macos-latest"
                 architecture:
                     - x64
                     - x86
-
                 exclude:
                     # Linux and macOS don't have x86 python
                     - os: "ubuntu-latest"
                       architecture: x86
                     - os: "macos-latest"
                       architecture: x86
-                    # Building on PyPy3 on Windows is broken
-                    - os: "windows-latest"
-                      py: "pypy3"
 
         name: "Python: ${{ matrix.py }}-${{ matrix.architecture }} on ${{ matrix.os }}"
         runs-on: ${{ matrix.os }}

diff --git a/CHANGES.txt b/CHANGES.txt
@@ -1,6 +1,22 @@
 Next Release
 ------------
 
+Bugfix
+~~~~~~
+
+- Fixed an issue whereby ``BytesIO`` objects were not properly closed, and
+  thereby would not get cleaned up until garbage collection would get around to
+  it.
+
+  This led to potential for random memory spikes/memory issues, see
+  https://github.com/Pylons/waitress/pull/358 and
+  https://github.com/Pylons/waitress/issues/357 .
+
+  With thanks to Florian Schulze for testing/vaidating this fix!
+
+Features
+~~~~~~~~
+
 - Add REQUEST_URI to the WSGI environment.
 
   REQUEST_URI is similar to ``request_uri`` in nginx. It is a string that

diff --git a/src/waitress/buffers.py b/src/waitress/buffers.py
@@ -234,11 +234,23 @@ def _create_buffer(self):
         return buf
 
     def _set_small_buffer(self):
-        self.buf = BytesIOBasedBuffer(self.buf)
+        oldbuf = self.buf
+        self.buf = BytesIOBasedBuffer(oldbuf)
+
+        # Attempt to close the old buffer
+        if hasattr(oldbuf, "close"):
+            oldbuf.close()
+
         self.overflowed = False
 
     def _set_large_buffer(self):
-        self.buf = TempfileBasedBuffer(self.buf)
+        oldbuf = self.buf
+        self.buf = TempfileBasedBuffer(oldbuf)
+
+        # Attempt to close the old buffer
+        if hasattr(oldbuf, "close"):
+            oldbuf.close()
+
         self.overflowed = True
 
     def append(self, s):

diff --git a/src/waitress/trigger.py b/src/waitress/trigger.py
@@ -131,7 +131,6 @@ def _close(self):
         def _physical_pull(self):
             os.write(self.trigger, b"x")
 
-
 else:  # pragma: no cover
     # Windows version; uses just sockets, because a pipe isn't select'able
     # on Windows.

diff --git a/tests/test_wasyncore.py b/tests/test_wasyncore.py
@@ -405,7 +405,6 @@ def _waitfor(func, pathname, waitall=False):
     def _unlink(filename):
         _waitfor(os.unlink, filename)
 
-
 else:
     _unlink = os.unlink