Misbehavior of BufferedRandom.write with raw file in append mode

[embray]

BPO	20082
Nosy	@pitrou, @benjaminp, @embray, @hynek, @serhiy-storchaka, @nitishch
PRs	bpo-20082: fix misbehavior of buffered writes to raw files in append mode #21729
Files	buffered-append-1.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-12-27.23:33:12.824>
labels = ['type-bug', 'expert-IO']
title = 'Misbehavior of BufferedRandom.write with raw file in append mode'
updated_at = <Date 2020-08-04.11:02:17.371>
user = 'https://github.com/embray'

bugs.python.org fields:

activity = <Date 2020-08-04.11:02:17.371>
actor = 'erik.bray'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['IO']
creation = <Date 2013-12-27.23:33:12.824>
creator = 'erik.bray'
dependencies = []
files = ['33282']
hgrepos = []
issue_num = 20082
keywords = ['patch']
message_count = 2.0
messages = ['207015', '314448']
nosy_count = 7.0
nosy_names = ['pitrou', 'benjamin.peterson', 'stutzbach', 'erik.bray', 'hynek', 'serhiy.storchaka', 'nitishch']
pr_nums = ['21729']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20082'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[embray]

In bpo-18876 I pointed out the following issue:

BufferedWriter/Random doesn't know the raw file was opened with O_APPEND so the writes it shows in the buffer differ from what will actually end up in the file. For example:

>>> f = open('test', 'wb')
>>> f.write(b'testest')
7
>>> f.close()
>>> f = open('test', 'ab+')
>>> f.tell()
7
>>> f.write(b'A')
1
>>> f.seek(0)
0
>>> f.read()
b'testestA'
>>> f.seek(0)
0
>>> f.read(1)
b't'
>>> f.write(b'B')
1
>>> f.seek(0)
0
>>> f.read()
b'tBstestA'
>>> f.flush()
>>> f.seek(0)
0
>>> f.read()
b'testestAB'

In this example, I read 1 byte from the beginning of the file, then write one byte. Because of O_APPEND, the effect of the write() call on the raw file is to append, regardless of where BufferedWriter seeks it to first. But before the f.flush() call f.read() just shows what's in the buffer which is not what will actually be written to the file. (Naturally, unbuffered io does not have this particular problem.)

Now that bpo-18876 we can test if a file was opened in append mode and correct for this. The attach patch includes a pretty simple solution that manually calls buffered_seek at the beginning of bufferedwriter_write if the raw file is in append mode. In doing so it made sense to split buffered_seek into two separate functions.

This might be overkill, however.

[embray]

I keep forgetting about this (given that it's like 5 years old now). Let me see if I can make a new PR for it...