Distutils generates the wrong export symbol for unicode module names

[da-woods]

BPO	39432
Nosy	@scoder, @ned-deily, @merwok, @encukou, @ambv, @zooba, @dstufft, @miss-islington, @da-woods
PRs	bpo-39432: Implement PEP-489 algorithm for non-ascii "PyInit_*" symbol names in distutils #18150 [3.8] bpo-39432: Implement PEP-489 algorithm for non-ascii "PyInit_*" symbol names in distutils (GH-18150) #18546 [3.8] bpo-39555: Fix distutils test to handle _d suffix on Windows debug build (GH-18357) #18548

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-02-18.13:17:31.345>
created_at = <Date 2020-01-23.13:48:15.580>
labels = ['3.8', 'type-bug', 'library', '3.9']
title = 'Distutils generates the wrong export symbol for unicode module names'
updated_at = <Date 2021-04-20.09:24:11.553>
user = 'https://github.com/da-woods'

bugs.python.org fields:

activity = <Date 2021-04-20.09:24:11.553>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-02-18.13:17:31.345>
closer = 'scoder'
components = ['Distutils']
creation = <Date 2020-01-23.13:48:15.580>
creator = 'da-woods'
dependencies = []
files = []
hgrepos = []
issue_num = 39432
keywords = ['patch']
message_count = 12.0
messages = ['360555', '361352', '361416', '361434', '361435', '361436', '361437', '361460', '361608', '361800', '362206', '391314']
nosy_count = 9.0
nosy_names = ['scoder', 'ned.deily', 'eric.araujo', 'petr.viktorin', 'lukasz.langa', 'steve.dower', 'dstufft', 'miss-islington', 'da-woods']
pr_nums = ['18150', '18546', '18548']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39432'
versions = ['Python 3.8', 'Python 3.9']

[da-woods]

Distuitls generates "export symbols" for extension modules to help ensure that they have have the correct linkage on Windows.

cpython/Lib/distutils/command/build_ext.py

Line 692 in 0d30ae1

initfunc_name = "PyInit_" + ext.name.split('.')[-1]

It generates the correct symbol in most causes, but if the filename contains unicode characters then it creates the wrong symbol, causing linkage errors.

The behaviour should be updated to reflect PEP-489: https://www.python.org/dev/peps/pep-0489/#export-hook-name

[miss-islington]

New changeset 9538bc9 by Stefan Behnel in branch 'master':
bpo-39432: Implement PEP-489 algorithm for non-ascii "PyInit_*" symbol names in distutils (GH-18150)
9538bc9

[scoder]

Ok, this is merged into 3.9. To which versions should we backport it?
Definitely 3.8, definitely not 3.5, probably not 3.6 (since it's not a security issue). Ned, what about 3.7?

[vstinner]

The test fails on Windows. Example on AMD64 Windows8.1 Refleaks 3.x:
https://buildbot.python.org/all/#/builders/157/builds/76

======================================================================
FAIL: test_unicode_module_names (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\distutils\tests\test_build_ext.py", line 315, in test_unicode_module_names
    self.assertRegex(cmd.get_ext_filename(modules[0].name), r'foo\..*')
AssertionError: Regex didn't match: 'foo\\..*' not found in 'foo_d.cp39-win_amd64.pyd'

======================================================================
FAIL: test_unicode_module_names (distutils.tests.test_build_ext.ParallelBuildExtTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\distutils\tests\test_build_ext.py", line 315, in test_unicode_module_names
    self.assertRegex(cmd.get_ext_filename(modules[0].name), r'foo\..*')
AssertionError: Regex didn't match: 'foo\\..*' not found in 'foo_d.cp39-win_amd64.pyd'

[vstinner]

Python 3.6 doesn't accept bugfixes anymore:
https://devguide.python.org/#status-of-python-branches

The bugfix can go into 3.7 and 3.8.