Accept mutable bytes-like objects

[serhiy-storchaka]

BPO	23001
Nosy	@pitrou, @vstinner, @ethanfurman, @vadmium, @serhiy-storchaka, @MojoVampire
Files	accept_mutable_buffers.patch accept_mutable_buffers_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2015-03-20.07:06:26.080>
created_at = <Date 2014-12-06.14:25:06.583>
labels = ['extension-modules', 'type-feature']
title = 'Accept mutable bytes-like objects'
updated_at = <Date 2015-03-20.10:01:21.895>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2015-03-20.10:01:21.895>
actor = 'vstinner'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2015-03-20.07:06:26.080>
closer = 'serhiy.storchaka'
components = ['Extension Modules']
creation = <Date 2014-12-06.14:25:06.583>
creator = 'serhiy.storchaka'
dependencies = []
files = ['37373', '38555']
hgrepos = []
issue_num = 23001
keywords = ['patch', 'needs review']
message_count = 14.0
messages = ['232244', '232246', '232252', '232255', '232264', '233438', '238502', '238598', '238607', '238608', '238628', '238630', '238633', '238635']
nosy_count = 8.0
nosy_names = ['pitrou', 'vstinner', 'Arfrever', 'ethan.furman', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'josh.r']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23001'
versions = ['Python 3.5']

[serhiy-storchaka]

Some builtins accept only read-only bytes-like objects (PyArg_Parse format codes "s#", "z#", "y", and "y#"). Proposed patch makes them accepting also mutable bytes-like objects such as bytearray.

I'm not sure that all these changes are useful, but in some cases this can get rid of copying binary data created in bytearray (e.g. read by readinto()).

[MojoVampire]

In the event of calls back into Python code in multithreaded execution (or GIL release), this would mean you no longer have guarantees as to the contents (or even the validity) of the pointer you get back. I'd think the only safe way to accept mutable buffers would be to use the s*, z*, y* codes, which lock the buffer to prevent resize/destruction. Do we want to open segfault vulnerabilities in arbitrary functions?

[serhiy-storchaka]

That is was the patch does. Convert from s# to s* etc.

See also bpo-22896 about potential bugs with the use of pointers to unlocked
buffers.

[MojoVampire]

Ah, sorry. Should have examined patch. I thought you were making a change to the behavior of s#, z#, y and y#, not converting actual uses of them. Again, sorry.

[pitrou]

+1 on the principle. I haven't looked at the patch.

[vadmium]

If this goes ahead, it would be nice adding notes to the documentation saying that bytearray() or whatever was previously not supported. There are APIs in Python 2.6 that had similar treatment with no documentation updates, and I keep being bitten by it.

[serhiy-storchaka]

Added versionchanged directives and fixed other documentation.

[vadmium]

I had a closer look at the changes, and most of them seem good. However I think I found one leak in fcntl(); see Reitveld.

I noticed there is no test for “ossaudiodev”. Would that be too hard, or is it just an oversight?

[serhiy-storchaka]

Thank you for your review Martin. Yes, the problem with fcntl() is not so easy and I'll left it for the separate issue.

I noticed there is no test for “ossaudiodev”. Would that be too hard, or is it just an oversight?

It is hard because test_ossaudiodev is not designed to just apply the test with different type of data, and this test doesn't work on my computer at all (no /dev/dsp).

[python-dev]

New changeset 17c77938c4e2 by Serhiy Storchaka in branch 'default':
Issue bpo-23001: Few functions in modules mmap, ossaudiodev, socket, ssl, and
https://hg.python.org/cpython/rev/17c77938c4e2

[python-dev]

New changeset 11548e1ac920 by Victor Stinner in branch 'default':
Issue bpo-23001: Fix typo
https://hg.python.org/cpython/rev/11548e1ac920

[python-dev]

New changeset d478a2a5738a by Victor Stinner in branch 'default':
Issue bpo-23709, bpo-23001: ossaudiodev now uses Py_ssize_t for sizes instead of int
https://hg.python.org/cpython/rev/d478a2a5738a

[serhiy-storchaka]

Thanks Victor.

[vstinner]

Thanks Victor.

You're welcome, I just worked recently on ossaudiodev for _Py_read/write functions functions. But I'm not able to run ossaudiodev neither :-( I hope that the test runs on some buildbots (FreeBSD?).