PyUnicode_AsUTF8AndSize Sometimes Segfaults With Incomplete Surrogate Pair

[willayd]

BPO	39113
Nosy	@serhiy-storchaka, @WillAyd
Files	testmodule.c: Extension Module For Use in Identifying Segfault

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-12-21.16:48:44.894>
created_at = <Date 2019-12-21.03:32:54.690>
labels = ['invalid']
title = 'PyUnicode_AsUTF8AndSize Sometimes Segfaults With Incomplete Surrogate Pair'
updated_at = <Date 2019-12-21.16:48:44.893>
user = 'https://github.com/willayd'

bugs.python.org fields:

activity = <Date 2019-12-21.16:48:44.893>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2019-12-21.16:48:44.894>
closer = 'serhiy.storchaka'
components = []
creation = <Date 2019-12-21.03:32:54.690>
creator = 'william.ayd'
dependencies = []
files = ['48798']
hgrepos = []
issue_num = 39113
keywords = []
message_count = 3.0
messages = ['358755', '358757', '358762']
nosy_count = 2.0
nosy_names = ['serhiy.storchaka', 'william.ayd']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue39113'
versions = []

[willayd]

With the attached extension module, if I run the following in the REPL:

>>> import libtest
>>>
>>> libtest.error_if_not_utf8("foo")
'foo'
>>> libtest.error_if_not_utf8("\ud83d")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 0: surrogates not allowed
>>> libtest.error_if_not_utf8("foo")
'foo'

Things seem OK. But the next invocation of

>> libtest.error_if_not_utf8("\ud83d")

Then causes a segfault. Note that the order of the input seems important; simply repeating the call with the invalid surrogate doesn't cause the segfault

[serhiy-storchaka]

Your function returns a borrowed reference. It xiuld cause ceash even without calling PyUnicode_AsUTF8AndSize. Add Py_INCREF(str)

[willayd]

Hmm my mistake - thanks!