File truncate() not defaulting to current position as documented

[fornax]

BPO	26158
Nosy	@pitrou, @benjaminp, @socketpair, @vadmium, @serhiy-storchaka, @eryksun, @zooba

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-01-19.15:19:44.966>
labels = ['type-bug', '3.8', '3.9', 'expert-IO', '3.10', 'docs']
title = 'File truncate() not defaulting to current position as documented'
updated_at = <Date 2021-02-26.17:25:06.794>
user = 'https://bugs.python.org/fornax'

bugs.python.org fields:

activity = <Date 2021-02-26.17:25:06.794>
actor = 'eryksun'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'IO']
creation = <Date 2016-01-19.15:19:44.966>
creator = 'fornax'
dependencies = []
files = []
hgrepos = []
issue_num = 26158
keywords = []
message_count = 15.0
messages = ['258600', '258611', '258612', '258613', '258614', '258615', '258617', '258618', '258619', '258622', '258624', '258626', '258634', '258637', '258639']
nosy_count = 12.0
nosy_names = ['pitrou', 'benjamin.peterson', 'stutzbach', 'Zero', 'docs@python', 'socketpair', 'martin.panter', 'serhiy.storchaka', 'eryksun', 'steve.dower', 'random832', 'fornax']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26158'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[fornax]

io.IOBase.truncate() documentation says:
"Resize the stream to the given size in bytes (or the current position if size is not specified). The current stream position isn’t changed. This resizing can extend or reduce the current file size. In case of extension, the contents of the new file area depend on the platform (on most systems, additional bytes are zero-filled). The new file size is returned."

However:
>>> open('temp.txt', 'w').write('ABCDE\nFGHIJ\nKLMNO\nPQRST\nUVWXY\nZ\n')
32
>>> f = open('temp.txt', 'r+')
>>> f.readline()
'ABCDE\n'
>>> f.tell()
6                   # As expected, current position is 6 after the readline
>>> f.truncate()
32                  # ?!

Verified that the document does not get truncated to 6 bytes as expected. Adding an explicit f.seek(6) before the truncate causes it to work properly (truncate to 6). It also works as expected using a StringIO rather than a file, or in Python 2 (used 2.7.9).

Tested in 3.4.3/Windows, 3.4.1/Linux, 3.5.1/Linux.

[serhiy-storchaka]

This is because the file is buffered.

>>> open('temp.txt', 'w').write('ABCDE\nFGHIJ\nKLMNO\nPQRST\nUVWXY\nZ\n')
32
>>> f = open('temp.txt', 'r+')
>>> f.readline()
'ABCDE\n'
>>> f.tell()
6
>>> f.buffer.tell()
32
>>> f.buffer.raw.tell()
32

The documentation needs a clarification.

[fornax]

To clarify... the intended behavior is for truncate to default to the current position of the buffer, rather than the current position as reported directly from the stream by tell?

That seems... surprising.

[eryksun]

Serhiy, why doesn't truncate do a seek(0, SEEK_CUR) to synchronize the buffer's file pointer before calling its truncate method? This also affects writing in "+" modes when the two file pointers are out of sync.

[serhiy-storchaka]

This is not always possible. Consider following example:

>>> open('temp.txt', 'wb').write(b'+BDAEMQQyBDMENA-')
16
>>> f = open('temp.txt', 'r+', encoding='utf-7')
>>> f.read(2)
'аб'

What should be the result of truncating?

I think it would be better to not implement truncate() for text files at all.

[fornax]

Heh... building on Serhiy's example:

>>> f.tell()
680564735109527527154978616360239628288

I'm way out of my depth here. The results seem surprising, but I lack the experience to be able to say what they "should" look like. So I guess if it's working as intended and just needs clarification in the documentation, so be it.

Thanks for pointing out what's actually causing the behavior.

[fornax]

After taking a little time to let this sink in, I'm going to play Devil's Advocate just a little more.

It sounds like you're basically saying that any read-write text-based modes (e.g. r+, w+) should be used at your own peril. While I understand your UTF-7 counterexample, and it's a fair point, is it out of line to expect that for encodings that operate on full bytes, file positioning should work a bit more intuitively? (Which is to say, a write/truncate after a read should take place in the position immediately following the end of the read.)

[fornax]

Another surprising result:

>>> open('temp.txt', 'w').write('ABCDE\nFGHIJ\nKLMNO\nPQRST\nUVWXY\nZ\n')
32
>>> f = open('temp.txt', 'r+')
>>> f.write('test')
4
>>> f.close()
>>> open('temp.txt').read()
'testE\nFGHIJ\nKLMNO\nPQRST\nUVWXY\nZ\n'

>>> open('temp.txt', 'w').write('ABCDE\nFGHIJ\nKLMNO\nPQRST\nUVWXY\nZ\n')
32
>>> f = open('temp.txt', 'r+')
>>> f.write('test')
4
>>> f.read(1)
'A'
>>> f.close()
>>> open('temp.txt').read()
'ABCDE\nFGHIJ\nKLMNO\nPQRST\nUVWXY\nZ\ntest'

The position of the write in the file depends on whether or not there is a subsequent read.

[serhiy-storchaka]

May be. Looking at the code, both Python and C implementations of TextIOWrapper look incorrect.

Python implementation:

    def truncate(self, pos=None):
        self.flush()
        if pos is None:
            pos = self.tell()
        return self.buffer.truncate(pos)

If pos is not specified, self.tell() is used as truncating position for underlying binary file. But self.tell() is not an offset in bytes, as seen from UTF-7 example. This is complex cookie that includes starting position in binary file, a number of bytes that should be read and feed to the decoder, and other decoder flags. Needed at least unpack the cookie returned by self.tell(), and raise an exception if it doesn't ambiguously point to binary file position.

C implementation is equivalent to:

    def truncate(self, pos=None):
        self.flush()
        return self.buffer.truncate(pos)

It just ignores decoder buffer.

[vadmium]

In theory, TextIOWrapper could rewrite the last bit of the file (or the whole file) to have the requested number of characters. But I wonder if it is worth it; maybe deprecation is better. Do you have a use case for any of these bugs, or are you just playing around to see what the methods do?

In bpo-12922, seek() and tell() were (re-)defined for TextIOBase, but the situation with truncate() was apparently not considered.

Perhaps the write()–read() bug is related to bpo-12215.

[fornax]

I don't have a specific use case. This spawned from a tangentially-related StackOverflow question (http://stackoverflow.com/questions/34858088), where in the answers a behavior difference between Python 2 and 3 was noted. I couldn't find any documentation to explain it, so I opened a follow-up question (http://stackoverflow.com/questions/34879318), and based on some feedback I got there, I opened up this issue.

Just to be sure I understand, you're suggesting deprecating truncate on text-mode file objects? And the interleaved read-writes are likely related to an issue that will be dealt with elsewhere?