shadows around the io truncate() semantics

[pakal]

BPO	6939
Nosy	@ncoghlan, @pitrou, @pakal
Files	PY27_truncate.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pitrou'
closed_at = <Date 2010-01-31.23:24:00.295>
created_at = <Date 2009-09-18.13:21:52.144>
labels = ['type-bug', 'release-blocker', 'expert-IO', 'docs']
title = 'shadows around the io truncate() semantics'
updated_at = <Date 2010-05-29.03:46:15.858>
user = 'https://github.com/pakal'

bugs.python.org fields:

activity = <Date 2010-05-29.03:46:15.858>
actor = 'ncoghlan'
assignee = 'pitrou'
closed = True
closed_date = <Date 2010-01-31.23:24:00.295>
closer = 'pitrou'
components = ['Documentation', 'IO']
creation = <Date 2009-09-18.13:21:52.144>
creator = 'pakal'
dependencies = []
files = ['16030']
hgrepos = []
issue_num = 6939
keywords = ['patch']
message_count = 27.0
messages = ['92822', '92827', '92833', '97273', '97274', '97277', '97321', '97326', '97328', '97329', '97834', '97836', '97849', '97853', '97854', '97855', '97873', '97911', '97966', '98301', '98450', '98451', '98479', '98632', '98635', '98673', '106707']
nosy_count = 3.0
nosy_names = ['ncoghlan', 'pitrou', 'pakal']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue6939'
versions = ['Python 3.1', 'Python 3.2']

[pakal]

Hello

I'm having trouble around the behaviour of the io module's truncate
methods, in py3k. If I remember well, under unix and older versions of
Python (with other file types), truncate never move the fiel pointer
(and had to fake that behaviour under windows as well, by
saving/restoring the file pointer).

However, I see nothing in the documentation of the io module about that,
and studying the _fileio.c code, it seems that now the file pointer is
(under all OS) moved to the truncation point, and left there.

Maybe it requires discusions on the mailing list, but personally I think
that the common unix behaviour (letting the file pointer untouched)
would be better for everyone, and that the doc should claim it.

Also, additional notes about the behaviour of that truncation (reminding
people that on some OS, the padding is zero-filled, on others it's
not...) would be great.

Current doc of io in py3k :
http://docs.python.org/py3k/library/io.html#io.IOBase.truncate
truncate(size=None)
Truncate the file to at most size bytes. size defaults to the
current file position, as returned by tell().

Cheers
Pascal

[pitrou]

I can't tell you why it was decided to behave like this. Perhaps it was
felt that, since FileIO is low-level (lower than the file object in
2.x), it shouldn't have to remember and restore the old file position.
That argument wouldn't apply to BufferedWriter, though.

I'm afraid I don't know anything about the padding differences myself.
If you give us more precisions on the subject, we will be able to
include them in the documentation.

[pakal]

Well, I guess it deserve discussion on the pydev mailing lits, that's
imo a rather important point, to be documented precisely.

Concerning the padding, I guess the semantic doesn't change between the
io module and the old file type, i.e :

file.truncate([size])¶
Truncate the file’s size. If the optional size argument is present,
the file is truncated to (at most) that size. The size defaults to the
current position. The current file position is not changed. Note that if
a specified size exceeds the file’s current size, the result is
platform-dependent: possibilities include that the file may remain
unchanged, increase to the specified size as if zero-filled, or increase
to the specified size with undefined new content. Availability: Windows,
many Unix variants.

Having platform-dependent semantics for a so important type is anyway
dangerous, I feel. The io module should be rock-solid, with very deeply
documented semantics, and implementations that force platforms to
conform to these specifications. Eg. I've noticed that all io.FileIO
methods raised IOErrors in case of problem, except in one case (wrapping
a bad file decsriptor) where it's OSError - that little thing might make
big programs crash in an ugly way, if they only caught IOError.
I'm currently working on a cross-platform (linux, mac, windows at least)
FileIO type, with very strict semantics and extended abilities (file
range shared and exclusive locking, with timeouts) ; even though it'll
initially be slower than _fileio (most of it is in pure python), maybe
it might solve most of these io module problems. I'll keep people informed.

[pakal]

Hello

I'm currently finalizing the API of my raw io file implementation, but I still have trouble with the trunk implementation of IOBase.truncate().

If I remember well, in the mailing list topic on this subject, GvR noted that this change of behaviour compared to python 2.x was not intended, and that it would be better to get back to the expected behaviour - not touching the file pointer - and to document the method in this way.

Are there new elements, advocating a status quo on this matter ?
Or shouldn't we add the portable_lseek() call in fileio.c to fix that ?

On a separate note, I'm confused about the "at most" phrase in the current documentation :
---

truncate(size=None)
    Truncate the file to at most size bytes. size defaults to the current file position, as returned by tell()

---

According to what I've read so far, a succesful truncate() call will always extend/reduce the file until teh desired size, isn't that so on all platforms ?

Regards
Pascal

[pitrou]

Are there new elements, advocating a status quo on this matter ?

Nothing, it's just lacking a patch from someone interested in the
matter.

On a separate note, I'm confused about the "at most" phrase in the
current documentation :
---
truncate(size=None)
Truncate the file to at most size bytes. size defaults to the
current file position, as returned by tell()
---
According to what I've read so far, a succesful truncate() call will
always extend/reduce the file until teh desired size, isn't that so on
all platforms ?

I suppose it was worded that way because some platforms may not support
extending a file when a parameter larger than the current size is given.
I don't know if such systems are widespread.

[pakal]

Allright, I'll try to work on it as soon as I manage to gather a decent compilation environment on windows...

[pakal]

Hi

Here is a patch for the python2.6 _fileio.c module, as well as the corresponding testcase.
I'll check the _pyio and C _io trunk modules asap.

[pakal]

Whoups - I forgot to bugfix as well the _bytesio classes... let's just forget about the previous patch.

[pakal]

Here is the new patch for py2.6, fixing (hopefully) all the truncate() methods.
Note that the python _BytesIO implementation isn't covered by the test suite, as it's shadowed by the C implementation ; but imo we shouldn't care : I've manually tested it once, and anyway the trunk sources don't have this problem.

[pakal]

And here is the python trunk patch, covering _Pyio and _io modules (+ corresponding tests).

[pakal]

Is there anything I can do to help this patch making its way to the trunk ?
I guess it'd be better if Python2.7 benefited from it, so that users don't run anymore the risk of relying of this undocumented and non-canonical "truncate" behaviour.

Regards,
Pascal