io close() swallowing exceptions

[pakal]

BPO	7865
Nosy	@amauryfa, @pitrou, @vstinner, @giampaolo, @devdanzin, @pakal
Files	test_error_close.py no_swallow_on_close3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-05-03.16:59:08.534>
created_at = <Date 2010-02-06.12:27:06.807>
labels = ['type-bug', 'expert-IO']
title = 'io close() swallowing exceptions'
updated_at = <Date 2010-05-06.21:07:11.349>
user = 'https://github.com/pakal'

bugs.python.org fields:

activity = <Date 2010-05-06.21:07:11.349>
actor = 'pakal'
assignee = 'none'
closed = True
closed_date = <Date 2010-05-03.16:59:08.534>
closer = 'pitrou'
components = ['IO']
creation = <Date 2010-02-06.12:27:06.807>
creator = 'pakal'
dependencies = []
files = ['16147', '17132']
hgrepos = []
issue_num = 7865
keywords = ['patch']
message_count = 16.0
messages = ['98940', '102673', '102710', '102715', '103970', '104121', '104143', '104351', '104415', '104426', '104441', '104460', '104577', '104579', '104854', '105165']
nosy_count = 6.0
nosy_names = ['amaury.forgeotdarc', 'pitrou', 'vstinner', 'giampaolo.rodola', 'ajaksu2', 'pakal']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue7865'
versions = ['Python 2.6', 'Python 3.1', 'Python 2.7', 'Python 3.2']

[pakal]

The current semantic of io streams is to swallow IOErrors on close(), eg. in _pyio from the trunk, we have each time constructs like:
try:
self.flush()
except IOError:
pass # If flush() fails, just give up

and in C files :
        /* If flush() fails, just give up */
        if (PyErr_ExceptionMatches(PyExc_IOError))
            PyErr_Clear();

I'd rather advocate letting exceptions flow, as users have the right to know if their io operations lost bytes.
Below is a test case for these swallowed exceptions.

PS : issues like http://bugs.python.org/issue7640 are actually much more crucial, so we shall let them higher priority

[devdanzin]

Wouldn't this break code that currently works?

[pakal]

Well, it would break code which currently ignores that it fails, so it's more a benefit than a loss for programmers imo.

I doubt the impact will be important though, because the io module is still quite recent, and furthermore errors on the last flush are quite unlikely to happen if previous ones succeeded (except "disk full", I don't see any reason for this to happen).

[pitrou]

You're right that silencing IO errors is bad. Patch welcome.

[pakal]

Patch and test to stop swallowing exceptions on stream close(), for python SVN trunk.

[pitrou]

The tests should probably check all three types of I/O (raw, buffered, text).

[pakal]

SHould be better this way then B-)

[pitrou]

I just tried the patch. One problem is that you are supposed to be able to call close() several times without having it fail:

>>> f = open("LICENSE")
>>> f.close()
>>> f.close()
>>> f = io.open("LICENSE")
>>> f.close()
>>> f.close()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: I/O operation on closed file.

This means your patch should be a little smarter.

[pakal]

I'm quite surprised it wasn't already covered by the test suite :S

Anyway I'm quite confused about the semantic which is expected from IO operations...

Should a flush on a closed stream fail (at the moment sometimes it does, sometimes doesn't) ? Why is sometimes ValueError used when I'd rather expect an IOError ?

[pitrou]

I'm quite surprised it wasn't already covered by the test suite :S

Probably an oversight. Do you want to add some tests?

Should a flush on a closed stream fail (at the moment sometimes it
does, sometimes doesn't) ?

It probably should, yes.

Why is sometimes ValueError used when I'd rather expect an IOError ?

Because it's not an IO error at all. No I/O occurs. You are just using
the file wrongly (or the wrong file), hence the ValueError.

[pakal]

Probably an oversight. Do you want to add some tests?

That's WIP

Because it's not an IO error at all. No I/O occurs. You are just using
the file wrongly (or the wrong file), hence the ValueError.

Then when you try to wrap a non-readable stream into a readable buffered stream (like BufferedRWPair), it should raise a value error as well, but currently it's rather:
"""if not reader.readable(): raise IOError('"reader" argument must be readable.')"""

[pitrou]

For what it's worth, I documented the possibility to call close() several times in r80592.

Then when you try to wrap a non-readable stream into a readable
buffered stream (like BufferedRWPair), it should raise a value error as
well,

Good point. Unfortunately, it's now a bit late to change this.
(in any case, it's a programming error to do such things and therefore the user will have to fix his/her code, rather than trying to catch the exception)

[pakal]

Here is a code/test patch which *should* fix the multiple close() case, and ensure flush() raise an error on closed file.

All IO test suites pass on my win32 (except test_largefile that I skip due to lack of hdd space).

I've noticed that the detach() semantic was quite different between _pyio and _io, by the way: C code raises valueerror when we try to access the stream after detaching it, whereas python lets attribute errors be raised. But maybe this is not so important, as these are programming errors anyway.

One thing I'm still wondering : why couldn't we obtain these C extension by cythonizing _pyio ? Are there features that cython lacks, or optimization considerations I'm not aware of ? Cython-generated extensions seem soooo easier to maintain...

[pitrou]

But maybe this is not so important, as these are programming errors
anyway.

Agreed :)

One thing I'm still wondering : why couldn't we obtain these C
extension by cythonizing _pyio ? Are there features that cython lacks,
or optimization considerations I'm not aware of ? Cython-generated
extensions seem soooo easier to maintain...

Several reasons:

• we don't want to depend on an external tool such as cython; the interpreter and its most critical modules just need a C compiler and a reasonably standard C library
• the language cython implements is not Python: it is both a superset and (more annoyingly) a subset of Python
• cython will not allow us, I think, to do as many optimizations as we do in the C version of the io library; the algorithms used are not the same as in the Python version, since raw C allows some much more efficient constructs (especially for handling memory buffers)

[pitrou]

Thanks for the patch! It was committed in r80720 (trunk), r80721 (2.6), r80722 (py3k), r80723 (3.1).

[pakal]

Cool, thanks a lot B-)