Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-26175: Fix SpooledTemporaryFile IOBase abstract #3249

Closed
1 change: 1 addition & 0 deletions Doc/library/tempfile.rst
Expand Up @@ -116,6 +116,7 @@ The module defines the following user-callable items:

.. versionchanged:: 3.8
Added *errors* parameter.
Implemented :class:`io.IOBase` inteface.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since 3.8.0 has been released, this should be moved to a new versionchanged:: 3.9 block.



.. function:: TemporaryDirectory(suffix=None, prefix=None, dir=None)
Expand Down
23 changes: 19 additions & 4 deletions Lib/tempfile.py
Expand Up @@ -620,7 +620,7 @@ def TemporaryFile(mode='w+b', buffering=-1, encoding=None,
_os.close(fd)
raise

class SpooledTemporaryFile:
class SpooledTemporaryFile(_io.IOBase):
"""Temporary file wrapper, specialized to switch from BytesIO
or StringIO to a real file when it exceeds a certain size or
when a fileno is needed.
Expand Down Expand Up @@ -679,6 +679,9 @@ def __exit__(self, exc, value, tb):
def __iter__(self):
return self._file.__iter__()

def __del__(self):
Copy link

@pppery pppery Jun 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be added: deleting the SpooledTemporaryFile will null out the reference to the underline file, and therefor call its __del__

Copy link
Member

@vadmium vadmium Jun 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if you inherit the default IOBase.__del__ implementation, it will call close and defeat any ResourceWarning that might otherwise be emitted. Perhaps it is better to make __del__ do nothing, or set it to object.__del__. [Seems that last option doesn’t exist.]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree: the underlying file should not be explicitly deleted as this is not expected behaviour. I can imagine a situation where someone is deliberately holding a reference to the underlying file and they would not expect/want it to be closed until their own reference has fallen out of scope. I've changed the method to do nothing and added a docstring to reflect this.

Thanks for your feedback

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree: the underlying file should not be explicitly deleted as this is not expected behaviour

The doc says: """This function operates exactly as TemporaryFile() does, except [irrelevant differences]."""

And then, about TemporaryFile: """On completion of the context or destruction of the file object the temporary file will be removed from the filesystem."""

So it seems the underlying file should be deleted when the file object disappears.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point is that IOBase implements a __del__ which has some side-effects. Those side effects are not desirable here. Any underlying buffers being wrapped, e.g. the TemporaryFile or the BytesIO, will be gc'd and handled as they should.

I think this implementation is correct.

return self._file.__del__()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, calling the __del__ method directly like that is not correct. __del__ is only to be called when all external references are gone. Instead I would suggest something like the following:

    def __del__(self):
        if not self.closed:
            import warnings
            warnings.warn('unclosed file %r' % (self,), ResourceWarning,
                          stacklevel=2, source=self)
            self.close()

(this is adapted from FileIO.__del__ in Lib/pyio.py)


def close(self):
self._file.close()

Expand Down Expand Up @@ -725,14 +728,23 @@ def newlines(self):
def read(self, *args):
return self._file.read(*args)

def readable(self):
return self._file.readable()

def readinto(self, b):
return self._file.readinto(b)

def readline(self, *args):
return self._file.readline(*args)

def readlines(self, *args):
return self._file.readlines(*args)

def seek(self, *args):
self._file.seek(*args)
return self._file.seek(*args)

def seekable(self):
return self._file.seekable()

@property
def softspace(self):
Expand All @@ -743,18 +755,21 @@ def tell(self):

def truncate(self, size=None):
if size is None:
self._file.truncate()
return self._file.truncate()
else:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this else is not needed, PEP warning, might as well fix

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not necessary to remove the else. It does no harm.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I never said it was necessary, just bad style as it unnecessarily increased the code depth, take my comment as an FYI

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wanted to confirm @merwok - nothing needs to be changed here for approval?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I advise to not change the else line.

if size > self._max_size:
self.rollover()
self._file.truncate(size)
return self._file.truncate(size)

def write(self, s):
file = self._file
rv = file.write(s)
self._check(file)
return rv

def writable(self):
return self._file.writable()

def writelines(self, iterable):
file = self._file
rv = file.writelines(iterable)
Expand Down
31 changes: 31 additions & 0 deletions Lib/test/test_tempfile.py
Expand Up @@ -985,6 +985,22 @@ def test_basic(self):
f = self.do_create(max_size=100, pre="a", suf=".txt")
self.assertFalse(f._rolled)

def test_is_iobase(self):
# SpooledTemporaryFile should implement io.IOBase
self.assertIsInstance(self.do_create(), io.IOBase)

def test_iobase_interface(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those tests are not very interesting. It would be better to test the methods operate properly.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It literally delegates everything to the underlying file.

The existing tests should cover the behaviors. The new behaviors are intended to flesh out the API so that SpooledTemporaryFile, and these tests seem to cover that.

# SpooledTemporaryFile should implement the io.IOBase interface.
# IOBase does not declare read(), readinto(), or write(), but
# they should be considered part of the interface.
iobase_abstract = {'read', 'readinto', 'write'}
spooledtempfile_abstract = set(dir(tempfile.SpooledTemporaryFile))
missing_attributes = iobase_abstract - spooledtempfile_abstract
self.assertFalse(
missing_attributes,
'io.IOBase attributes missing from SpooledTemporaryFile'
)

def test_del_on_close(self):
# A SpooledTemporaryFile is deleted when closed
dir = tempfile.mkdtemp()
Expand All @@ -1000,6 +1016,21 @@ def test_del_on_close(self):
finally:
os.rmdir(dir)

def test_del_rolled_file(self):
# The rolled file should be deleted when the
# SpooledTemporaryFile object is deleted. This should raise a
# ResourceWarning since the file was not explicitly closed.
f = self.do_create(max_size=2)
f.write(b'foo')
filename = f.name
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this actually work? Here I get something like:

>>> f = tempfile.SpooledTemporaryFile()
>>> f.name
>>> f.rollover()
>>> f.name
3

self.assertTrue(os.path.exists(filename))
with self.assertWarns(ResourceWarning):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we also expect a ResourceWarning to be raised when the file is not rolled over? If yes, there should be a test for it. If not, then the __del__ implementation can be removed.

f.__del__()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not correct to call __del__ directly. Instead, use del f or f = None.

self.assertFalse(
os.path.exists(filename),
"Rolled SpooledTemporaryFile not deleted after __del__"
)

def test_rewrite_small(self):
# A SpooledTemporaryFile can be written to multiple within the max_size
f = self.do_create(max_size=30)
Expand Down
@@ -0,0 +1,2 @@
Fully implement io.IOBase interface for tempfile.SpooledTemporaryFile.
Patch by Gary Fernie.