PyArg_Parse*(): "z" should not accept bytes

[vstinner]

BPO	8949
Nosy	@malemburg, @loewis, @vstinner
Files	pyarg_z_no_bytes-2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-06-24.22:11:00.734>
created_at = <Date 2010-06-08.21:23:21.729>
labels = ['interpreter-core']
title = 'PyArg_Parse*(): "z" should not accept bytes'
updated_at = <Date 2010-06-24.22:11:00.732>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2010-06-24.22:11:00.732>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2010-06-24.22:11:00.734>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2010-06-08.21:23:21.729>
creator = 'vstinner'
dependencies = []
files = ['17661']
hgrepos = []
issue_num = 8949
keywords = ['patch']
message_count = 12.0
messages = ['107351', '107357', '107384', '107388', '107393', '107579', '107583', '107600', '107748', '108152', '108311', '108554']
nosy_count = 3.0
nosy_names = ['lemburg', 'loewis', 'vstinner']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue8949'
versions = ['Python 3.2']

[vstinner]

"z" is supposed to be the same format than "s" but accepts None.... Except that "z" does also accept bytes. I suppose that "s" was "fixed" between Python2 and Python3, but "z" is not fixed yet.

I think that it can be fixed in Python 3.2 without any deprecation process.

[vstinner]

Hum, the patch doesn't update the documentation (at least Porting section of the What's new in Python 3.2).

[malemburg]

"z" should not accept bytes... why not ?

"z" is the same as "s" with the addition that passing None
as parameter will result in the pointer to get set to NULL.

"s" accepts bytes via the buffer interface, so why should
"z" behave differently ?

If a function should only accept Unicode or None, then "Z"
should be used instead.

[vstinner]

Le mercredi 09 juin 2010 13:06:53, vous avez écrit :

"s" accepts bytes via the buffer interface

No. "s*" and "s#" do accept any buffer compatible object (including bytes and
bytearray), but "s" doesn't.

I fixed recently the documentation about that.

[malemburg]

STINNER Victor wrote:

STINNER Victor <victor.stinner@haypocalc.com> added the comment:

Le mercredi 09 juin 2010 13:06:53, vous avez écrit :
> "s" accepts bytes via the buffer interface

No. "s*" and "s#" do accept any buffer compatible object (including bytes and
bytearray), but "s" doesn't.

I fixed recently the documentation about that.

Hmm, that sounds like an oversight to me. Why should "s" not accept
bytes when "s#" and "s*" do ?

The only difference between "s" and "s#" is that you work with
NUL-terminated strings as opposed to binary or text data with embedded
NULs. Even Unicode objects can contain embedded NULs, so from that
perspective there's no difference.

[vstinner]

Some examples of functions using "s" format:

• str.encode(encoding, errors), bytes.decode(encoding, errors): both arguments have to be unicode strings
• compile(source, filename, mode, ...): filename and mode have to be unicode strings
• crypt.crypt(word, salt): both arguments have to be unicode strings

I think that crypt() should also accept bytes, but not str.encode() nor bytes.decode().

Some examples of functions using "z" format:

• _locale.bindtextdomain(domain, dirname): dirname uses "z" format and so accepts str, bytes or buffer compatible object. It should use PyUnicode_FSConverter() instead. But I agree that bytes is welcomed here.
• readline.(write_history_file|read_init_file|read_history_file) functions do use "z" to parse a filename. PyUnicode_FSConverter() would also be better, but in this case "z" is better than "s" :-)

I don't know why "s" and "z" are different about bytes, but it will be difficult to change it without changing a lot ot code (all functions using these formats). I tried to reject types different than str for "z": most tests of the test suite fail. I tried to accept bytes for "s" format: "unicode".encode(b'abc') does segfault.

[vstinner]

I tried to reject types different than str for "z":
most tests of the test suite fail

Wait, what? No. I modified the wrong line of code :-) The whole test suite pass without any error if "z" doesn't accept bytes anymore.

[vstinner]

I patched readline (r81918) and locale (r81914) modules to use PyUnicode_FSConverter(). They don't use "z" format anymore to parse a filename.

I checked last functions using "z" format (not "z#" or "z*", only "z") and I think that it's ok ("z" can be patched to reject bytes). You can get the full list using:

grep -n 'PyArg_Parse[^(]([^"]"[^";:]z[^#*]' */.c

--

About the difference between s/z and s*/s#/z*/z#: I don't understand why they are different, but I think that it's too late to change that. Too much functions (and third party modules?) suppose that s*/s#/z*/z# supports buffer compatible objects.

[vstinner]

Update the patch to update also the new tests that I added in r81973 (to fix bpo-8592).

[vstinner]

@lemburg: So what is your opinion on this issue?

[malemburg]

STINNER Victor wrote:

STINNER Victor <victor.stinner@haypocalc.com> added the comment:

@lemburg: So what is your opinion on this issue?

You're probably right: it's too late to change "s" to accept buffer
interface compatible objects as well.

In that case, symmetry is more important, so +1 on rejecting bytes
for "z" as well.

Please add a note to the NEWS file for this change.

[vstinner]

Patch commited to 3.2 (r82200), blocked in 3.1 (r82201).