The result type of bytearray formatting is not stable

[berkerpeksag]

BPO	26766
Nosy	@vstinner, @ethanfurman, @berkerpeksag, @serhiy-storchaka
Files	bytearray_mod.diff writer_bytearray_mod.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-04-15.22:20:34.426>
created_at = <Date 2016-04-15.09:43:21.037>
labels = ['interpreter-core', 'type-bug']
title = 'The result type of bytearray formatting is not stable'
updated_at = <Date 2016-04-15.22:20:34.425>
user = 'https://github.com/berkerpeksag'

bugs.python.org fields:

activity = <Date 2016-04-15.22:20:34.425>
actor = 'berker.peksag'
assignee = 'none'
closed = True
closed_date = <Date 2016-04-15.22:20:34.426>
closer = 'berker.peksag'
components = ['Interpreter Core']
creation = <Date 2016-04-15.09:43:21.037>
creator = 'berker.peksag'
dependencies = []
files = ['42466', '42473']
hgrepos = []
issue_num = 26766
keywords = ['patch']
message_count = 15.0
messages = ['263462', '263465', '263469', '263473', '263477', '263485', '263486', '263492', '263496', '263500', '263502', '263503', '263504', '263529', '263530']
nosy_count = 5.0
nosy_names = ['vstinner', 'ethan.furman', 'python-dev', 'berker.peksag', 'serhiy.storchaka']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26766'
versions = ['Python 3.6']

[berkerpeksag]

I noticed this while looking at bpo-26764.

bytearray_mod() and bytearray_format() both have checks for PyByteArray_Check(v). The check in bytearray_format() looks redundant to me.

Here is a patch.

[serhiy-storchaka]

Yes, I noticed this, but left for future (post-issue26765) refactoring.

The patch LGTM, but notice Victor's comment to bpo-26764.

[serhiy-storchaka]

And may be bytearray_mod in 3.6 is not correct. In 3.5 it returns bytearray, in 3.6 it returns bytes.

[berkerpeksag]

Do you have an example code? It returns bytearray for me in both 3.5 and 3.6. use_bytearray parameter of _PyBytes_FormatEx() is 1 in bytearray_mod().

[serhiy-storchaka]

Python 3.5:

>>> bytearray(b'%d') % 42
bytearray(b'42')

Python 3.6:

>>> bytearray(b'%d') % 42
b'42'

[berkerpeksag]

Thanks! More examples:

Python 3.6:

>>> bytearray(b'hello %b') % b"world"
bytearray(b'hello world')
>>> bytearray(b'hello %b') % b"wor"
b'hello wor'

Python 3.5:

>>> bytearray(b'hello %b') % b"world"
bytearray(b'hello world')
>>> bytearray(b'hello %b') % b"wor"
bytearray(b'hello wor')

[serhiy-storchaka]

Hmm, looks the result type is very unstable.

On release build:

>>> bytearray(b'hello %b') % b"world"
b'hello world'
>>> bytearray(b'hello %b') % b"wor"
b'hello wor'

On debug build:

>>> bytearray(b'hello %b') % b"world"
bytearray(b'hello world')
>>> bytearray(b'hello %b') % b"wor"
b'hello wor'

And current test_bytes.py is passed on release build, but is failed on debug build.

[vstinner]

It looks like a bug introduced by me in the issue bpo-25399.

bytearray%args must return a bytearray object.

I understand that test_bytes lacks tests since it missed the bug.

[vstinner]

Here is a fix for bytearray%args. As expected, it was a bug in the new _PyBytesWriter API (introduced in Python 3.6 to implementation optimizations).

[serhiy-storchaka]

LGTM, please commit. Why the behavior is vary in release and debug builds?

[python-dev]

New changeset 1eb586d5b321 by Victor Stinner in branch 'default':
Issue bpo-26766: Fix _PyBytesWriter_Finish()
https://hg.python.org/cpython/rev/1eb586d5b321

[vstinner]

LGTM, please commit.

Ok, done.

Why the behavior is vary in release and debug builds?

See _PyBytesWriter_Alloc(): the code is designed to detect bugs earlier in debug mode, it only uses 10 bytes of the "small buffer" (buffer allocated on the stack) rather than the full size of the smaller buffer (512 bytes).

It looks like this hack helped to find a real bug :-)

[vstinner]

@berker: bytearray_mod.diff LGTM, you can push it.

[python-dev]

New changeset b3e3efc5afa4 by Berker Peksag in branch 'default':
Issue bpo-26766: Remove redundant bytearray_format() from bytearrayobject.c
https://hg.python.org/cpython/rev/b3e3efc5afa4

[berkerpeksag]

Thanks for the reviews!