confusing BufferError: Existing exports of data: object cannot be re-sized

[arigo]

BPO	41097
Nosy	@pewscorner, @websurfer5
PRs	bpo-41097: allow io.BufferIO.write() operations when there are buffer views and no buffer resize is required #21792

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-06-24.07:29:18.464>
labels = ['interpreter-core', 'type-bug']
title = 'confusing BufferError: Existing exports of data: object cannot be re-sized'
updated_at = <Date 2020-08-09.06:12:38.141>
user = 'https://github.com/arigo'

bugs.python.org fields:

activity = <Date 2020-08-09.06:12:38.141>
actor = 'Jeffrey.Kintscher'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2020-06-24.07:29:18.464>
creator = 'arigo'
dependencies = []
files = []
hgrepos = []
issue_num = 41097
keywords = ['patch']
message_count = 4.0
messages = ['372237', '372286', '375061', '375063']
nosy_count = 2.0
nosy_names = ['pewscorner', 'Jeffrey.Kintscher']
pr_nums = ['21792']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41097'
versions = []

[arigo]

The behavior (tested in 3.6 and 3.9) of io.BytesIO().getbuffer() gives a unexpected exception message:

>>> b = io.BytesIO()
>>> b.write(b'abc')
3
>>> buf = b.getbuffer()
>>> b.seek(0)
0
>>> b.write(b'?')     # or anything up to 3 bytes
BufferError: Existing exports of data: object cannot be re-sized

The error message pretends that the problem is in resizing the BytesIO object, but the write() is not actually causing any resize.

I am not sure if the bug is a wrong error message (and all writes are supposed to be forbidden) or a wrongly forbidden write() (after all, we can use the buffer itself to write into the same area of memory).

[pewscorner]

I'm getting the same error (in 3.8.3) for this code:

o = bytearray(1)
o += o

There's no error if the second line is replaced with:

o += bytearray(1)

Not sure if this is related.

[websurfer5]

The error message is found in three different locations in the source code. One is in the bytearray class, and the other two are in the BytesIO class. They are unrelated code paths.

[websurfer5]

>>> import io
>>> b = io.BytesIO()
>>> b.write(b'abc')
3
>>> buf = b.getbuffer()
>>> b.seek(0)
0
>>> b.write(b'?')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
BufferError: Existing exports of data: object cannot be re-sized
>>> 

The problem is caused by the b.getbuffer() call. It increments a reference counter in the BytesIO object that causes the b.write() call to fail because the counter is > 0. The error message is misleading. The counter is decremented when the buffer view is deleted.

>>> import io
>>> b = io.BytesIO()
>>> b.write(b'abc')
3
>>> buf = b.getbuffer()
>>> b.seek(0)
0
>>> b.write(b'?')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
BufferError: Existing exports of data: object cannot be re-sized
>>> del buf
>>> b.write(b'?')
1
>>> b.getvalue()
b'?bc'
>>> 

The documentation for io.BytesIO.getbuffer() says "Note: As long as the view exists, the BytesIO object cannot be resized or closed." Either this is a bug, or the documentation needs to be updated to say the io.BytesIO object is unwritable while any buffer views exist.