bpo-41894: Fix UTF-8 errors loading native module #22466

kadler · 2020-09-30T17:42:01Z

When running in a non-UTF-8 locale, if an error occurs while importing a
native Python module (say because a dependent share library is missing),
the error message string returned may contain non-ASCII code points
causing a UnicodeDecodeError.

While this is most likely to occur on AIX, which still uses non-UTF-8
locales by default, it could occur on any Unix-like system.

https://bugs.python.org/issue41894

Python/dynload_shlib.c

taleinat · 2020-10-02T14:05:04Z

@aeros, why the "skip news" label? This is clearly a bug fix and those usually entail NEWS entries.

kadler · 2020-10-02T16:48:58Z

I added a NEWS entry.

Python/dynload_aix.c

serhiy-storchaka

LGTM

Misc/NEWS.d/next/Core and Builtins/2020-10-02-11-35-33.bpo-41894.ffmtOt.rst

Python/dynload_aix.c

When running in a non-UTF-8 locale, if an error occurs while importing a native Python module (say because a dependent share library is missing), the error message string returned may contain non-ASCII code points causing a UnicodeDecodeError. PyUnicode_DecodeFSDefault is used for buffers which may contain filesystem paths. For consistency with os.strerror(), PyUnicode_DecodeLocale is used for buffers which contain system error messages. While the shortname parameter is always encoded in ASCII according to PEP 489, it is left decoded using PyUnicode_FromString to minimize the changes and since it should not affect the decoding (albeit _potentially_ slower). In dynload_hpux, since the error buffer contains a message generated from a static ASCII string and the module filesystem path, PyUnicode_DecodeFSDefault is used instead of PyUnicode_DecodeLocale as is used elsewhere.

For both dynload_aix and dynload_hpux, properly handle the possibility that decoding strings may return NULL and when such an error happens, properly decrement any previously decoded strings and return early. In addition, in dynload_aix, ensure that we pass the decoded string *object* pathname_ob to PyErr_SetImportError instead of the original pathname buffer. Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

kadler · 2020-10-13T23:48:50Z

Ok, I believe all suggestions have been addressed. Wherever a filesystem path has been used, I have switched it to use PyUnicode_DecodeFSDefault and for any system error messages I have used PyUnicode_DecodeLocale. While PEP 489 dictates that the function name will always be ASCII, I have not switched to using PyUnicode_DecodeASCII since it does not solve any problem (other than being potentially less efficient), thus reducing the size of the changes.

serhiy-storchaka

LGTM.

miss-islington · 2020-10-15T01:53:30Z

Thanks @kadler for the PR, and @methane for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8, 3.9.
🐍🍒⛏🤖

…GH-22466) When running in a non-UTF-8 locale, if an error occurs while importing a native Python module (say because a dependent share library is missing), the error message string returned may contain non-ASCII code points causing a UnicodeDecodeError. PyUnicode_DecodeFSDefault is used for buffers which may contain filesystem paths. For consistency with os.strerror(), PyUnicode_DecodeLocale is used for buffers which contain system error messages. While the shortname parameter is always encoded in ASCII according to PEP 489, it is left decoded using PyUnicode_FromString to minimize the changes and since it should not affect the decoding (albeit _potentially_ slower). In dynload_hpux, since the error buffer contains a message generated from a static ASCII string and the module filesystem path, PyUnicode_DecodeFSDefault is used instead of PyUnicode_DecodeLocale as is used elsewhere. * bpo-41894: Fix bugs in dynload error msg handling For both dynload_aix and dynload_hpux, properly handle the possibility that decoding strings may return NULL and when such an error happens, properly decrement any previously decoded strings and return early. In addition, in dynload_aix, ensure that we pass the decoded string *object* pathname_ob to PyErr_SetImportError instead of the original pathname buffer. Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> (cherry picked from commit 2d2af32) Co-authored-by: Kevin Adler <kadler@us.ibm.com>

bedevere-bot · 2020-10-15T01:53:44Z

GH-22704 is a backport of this pull request to the 3.9 branch.

…GH-22466) When running in a non-UTF-8 locale, if an error occurs while importing a native Python module (say because a dependent share library is missing), the error message string returned may contain non-ASCII code points causing a UnicodeDecodeError. PyUnicode_DecodeFSDefault is used for buffers which may contain filesystem paths. For consistency with os.strerror(), PyUnicode_DecodeLocale is used for buffers which contain system error messages. While the shortname parameter is always encoded in ASCII according to PEP 489, it is left decoded using PyUnicode_FromString to minimize the changes and since it should not affect the decoding (albeit _potentially_ slower). In dynload_hpux, since the error buffer contains a message generated from a static ASCII string and the module filesystem path, PyUnicode_DecodeFSDefault is used instead of PyUnicode_DecodeLocale as is used elsewhere. * bpo-41894: Fix bugs in dynload error msg handling For both dynload_aix and dynload_hpux, properly handle the possibility that decoding strings may return NULL and when such an error happens, properly decrement any previously decoded strings and return early. In addition, in dynload_aix, ensure that we pass the decoded string *object* pathname_ob to PyErr_SetImportError instead of the original pathname buffer. Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> (cherry picked from commit 2d2af32) Co-authored-by: Kevin Adler <kadler@us.ibm.com>

bedevere-bot · 2020-10-15T01:53:52Z

GH-22705 is a backport of this pull request to the 3.8 branch.

When running in a non-UTF-8 locale, if an error occurs while importing a native Python module (say because a dependent share library is missing), the error message string returned may contain non-ASCII code points causing a UnicodeDecodeError. PyUnicode_DecodeFSDefault is used for buffers which may contain filesystem paths. For consistency with os.strerror(), PyUnicode_DecodeLocale is used for buffers which contain system error messages. While the shortname parameter is always encoded in ASCII according to PEP 489, it is left decoded using PyUnicode_FromString to minimize the changes and since it should not affect the decoding (albeit _potentially_ slower). In dynload_hpux, since the error buffer contains a message generated from a static ASCII string and the module filesystem path, PyUnicode_DecodeFSDefault is used instead of PyUnicode_DecodeLocale as is used elsewhere. * bpo-41894: Fix bugs in dynload error msg handling For both dynload_aix and dynload_hpux, properly handle the possibility that decoding strings may return NULL and when such an error happens, properly decrement any previously decoded strings and return early. In addition, in dynload_aix, ensure that we pass the decoded string *object* pathname_ob to PyErr_SetImportError instead of the original pathname buffer. Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> (cherry picked from commit 2d2af32) Co-authored-by: Kevin Adler <kadler@us.ibm.com>

…GH-22466) When running in a non-UTF-8 locale, if an error occurs while importing a native Python module (say because a dependent share library is missing), the error message string returned may contain non-ASCII code points causing a UnicodeDecodeError. PyUnicode_DecodeFSDefault is used for buffers which may contain filesystem paths. For consistency with os.strerror(), PyUnicode_DecodeLocale is used for buffers which contain system error messages. While the shortname parameter is always encoded in ASCII according to PEP 489, it is left decoded using PyUnicode_FromString to minimize the changes and since it should not affect the decoding (albeit _potentially_ slower). In dynload_hpux, since the error buffer contains a message generated from a static ASCII string and the module filesystem path, PyUnicode_DecodeFSDefault is used instead of PyUnicode_DecodeLocale as is used elsewhere. * bpo-41894: Fix bugs in dynload error msg handling For both dynload_aix and dynload_hpux, properly handle the possibility that decoding strings may return NULL and when such an error happens, properly decrement any previously decoded strings and return early. In addition, in dynload_aix, ensure that we pass the decoded string *object* pathname_ob to PyErr_SetImportError instead of the original pathname buffer. Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

the-knights-who-say-ni added the CLA signed label Sep 30, 2020

bedevere-bot added the awaiting review label Sep 30, 2020

aeros requested a review from methane October 2, 2020 00:34

aeros added the skip news label Oct 2, 2020

methane reviewed Oct 2, 2020

View reviewed changes

Python/dynload_shlib.c Show resolved Hide resolved

taleinat added the type-bug An unexpected behavior, bug, or error label Oct 2, 2020

kadler force-pushed the unicode-dynload-error branch from a061b90 to 159f723 Compare October 2, 2020 16:48

kadler force-pushed the unicode-dynload-error branch from 159f723 to 02dcee4 Compare October 2, 2020 17:13

methane added needs backport to 3.9 only security fixes and removed skip news labels Oct 4, 2020

ZackerySpytz reviewed Oct 5, 2020

View reviewed changes

Python/dynload_aix.c Show resolved Hide resolved

Python/dynload_aix.c Outdated Show resolved Hide resolved

serhiy-storchaka reviewed Oct 7, 2020

View reviewed changes

Python/dynload_aix.c Outdated Show resolved Hide resolved

kadler force-pushed the unicode-dynload-error branch from 398b987 to 46b331b Compare October 7, 2020 22:11

serhiy-storchaka approved these changes Oct 8, 2020

View reviewed changes

Misc/NEWS.d/next/Core and Builtins/2020-10-02-11-35-33.bpo-41894.ffmtOt.rst Outdated Show resolved Hide resolved

bedevere-bot added awaiting merge and removed awaiting review labels Oct 8, 2020

kadler force-pushed the unicode-dynload-error branch from 46b331b to fdb8782 Compare October 12, 2020 19:37

serhiy-storchaka reviewed Oct 12, 2020

View reviewed changes

Python/dynload_aix.c Outdated Show resolved Hide resolved

kadler and others added 2 commits October 13, 2020 18:36

kadler force-pushed the unicode-dynload-error branch from fdb8782 to 9230f48 Compare October 13, 2020 23:43

serhiy-storchaka approved these changes Oct 14, 2020

View reviewed changes

methane approved these changes Oct 15, 2020

View reviewed changes

methane added the needs backport to 3.8 label Oct 15, 2020

methane merged commit 2d2af32 into python:master Oct 15, 2020

bedevere-bot removed the awaiting merge label Oct 15, 2020

bedevere-bot removed the needs backport to 3.9 only security fixes label Oct 15, 2020

bedevere-bot removed the needs backport to 3.8 label Oct 15, 2020

kadler deleted the unicode-dynload-error branch October 16, 2020 15:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

bpo-41894: Fix UTF-8 errors loading native module #22466

bpo-41894: Fix UTF-8 errors loading native module #22466

Uh oh!

kadler commented Sep 30, 2020 •

edited by bedevere-bot

Loading

Uh oh!

Uh oh!

taleinat commented Oct 2, 2020

Uh oh!

kadler commented Oct 2, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

serhiy-storchaka left a comment

Uh oh!

Uh oh!

Uh oh!

kadler commented Oct 13, 2020

Uh oh!

serhiy-storchaka left a comment

Uh oh!

miss-islington commented Oct 15, 2020

Uh oh!

bedevere-bot commented Oct 15, 2020

Uh oh!

bedevere-bot commented Oct 15, 2020

Uh oh!

Uh oh!

Uh oh!

bpo-41894: Fix UTF-8 errors loading native module #22466

bpo-41894: Fix UTF-8 errors loading native module #22466

Uh oh!

Conversation

kadler commented Sep 30, 2020 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

taleinat commented Oct 2, 2020

Uh oh!

kadler commented Oct 2, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kadler commented Oct 13, 2020

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

miss-islington commented Oct 15, 2020

Uh oh!

bedevere-bot commented Oct 15, 2020

Uh oh!

bedevere-bot commented Oct 15, 2020

Uh oh!

Uh oh!

kadler commented Sep 30, 2020 •

edited by bedevere-bot

Loading