create_unicode_buffer() fails on non-BMP strings on Windows

[gergelyerdelyi]

BPO	19865
Nosy	@amauryfa, @abalkin, @vstinner, @meadori, @eryksun, @ZackerySpytz, @miss-islington
PRs	bpo-19865: ctypes.create_unicode_buffer() fails on non-BMP strings on Windows #14081 [3.7] bpo-19865: ctypes.create_unicode_buffer() supports non-BMP strings on Windows (GH-14081) #14087 [3.8] bpo-19865: ctypes.create_unicode_buffer() supports non-BMP strings on Windows (GH-14081) #14088

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-06-14.16:54:21.044>
created_at = <Date 2013-12-02.19:36:45.979>
labels = ['3.8', 'ctypes', 'type-bug', '3.7', '3.9']
title = 'create_unicode_buffer() fails on non-BMP strings on Windows'
updated_at = <Date 2019-06-14.16:54:21.043>
user = 'https://bugs.python.org/gergelyerdelyi'

bugs.python.org fields:

activity = <Date 2019-06-14.16:54:21.043>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-06-14.16:54:21.044>
closer = 'vstinner'
components = ['ctypes']
creation = <Date 2013-12-02.19:36:45.979>
creator = 'gergely.erdelyi'
dependencies = []
files = []
hgrepos = []
issue_num = 19865
keywords = ['patch']
message_count = 9.0
messages = ['205045', '228405', '228424', '330583', '345596', '345601', '345609', '345610', '345611']
nosy_count = 9.0
nosy_names = ['amaury.forgeotdarc', 'belopolsky', 'vstinner', 'meador.inge', 'eryksun', 'gergely.erdelyi', 'ZackerySpytz', 'miss-islington', 'Leonard de Ruijter']
pr_nums = ['14081', '14087', '14088']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19865'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[gergelyerdelyi]

create_unicode_buffer() fails on Windows if the initializer string contains unicode code points outside of the Basic Multilingual Plane and an explicit length is not specified.

The problem appears to be rooted in the fact that, since PEP-393, len() returns the number of code points, which does not always correspond to the number of 16-bit wchar words needed for the encoding on Windows. Because of that, the preallocated c_wchar buffer will be too short for the UTF-16 string.

The following small snippet demonstrates the problem:

from ctypes import create_unicode_buffer
b = create_unicode_buffer("\U00028318\U00028319")
print(b)

File "c:\Python33\lib\ctypes\init.py", line 294, in create_unicode_buffer
buf.value = init
ValueError: string too long

[BreamoreBoy]

I can confirm that this problem still exists so can someone take a look please, thanks.

[eryksun]

When sizeof(c_wchar) == 2, it can just count the number of non-BMP ordinals in the string. Another approach would be to use size = pythonapi.PyUnicode_AsWideChar(init, None, 0), but then the whole function may as well be implemented in the _ctypes extension module.

[LeonarddeRuijter]

I'm still able to reproduce this issue with ctypes under Python 3.7.0

[ZackerySpytz]

I have created a pull request for this issue. Please take a look.

[vstinner]

New changeset 9765efc by Victor Stinner (Zackery Spytz) in branch 'master':
bpo-19865: ctypes.create_unicode_buffer() supports non-BMP strings on Windows (GH-14081)
9765efc

[miss-islington]

New changeset 0b592d5 by Miss Islington (bot) in branch '3.7':
bpo-19865: ctypes.create_unicode_buffer() supports non-BMP strings on Windows (GH-14081)
0b592d5

[miss-islington]

New changeset b0f6fa8 by Miss Islington (bot) in branch '3.8':
bpo-19865: ctypes.create_unicode_buffer() supports non-BMP strings on Windows (GH-14081)
b0f6fa8

[vstinner]

Thanks Zackery Spytz for the fix. Thanks Gergely Erdélyi for the bug report! Sorry for the long delay.