[3.8] bpo-41894: Fix UnicodeDecodeError while loading native module (GH-22466) #22705
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…GH-22466) When running in a non-UTF-8 locale, if an error occurs while importing a native Python module (say because a dependent share library is missing), the error message string returned may contain non-ASCII code points causing a UnicodeDecodeError. PyUnicode_DecodeFSDefault is used for buffers which may contain filesystem paths. For consistency with os.strerror(), PyUnicode_DecodeLocale is used for buffers which contain system error messages. While the shortname parameter is always encoded in ASCII according to PEP 489, it is left decoded using PyUnicode_FromString to minimize the changes and since it should not affect the decoding (albeit _potentially_ slower). In dynload_hpux, since the error buffer contains a message generated from a static ASCII string and the module filesystem path, PyUnicode_DecodeFSDefault is used instead of PyUnicode_DecodeLocale as is used elsewhere. * bpo-41894: Fix bugs in dynload error msg handling For both dynload_aix and dynload_hpux, properly handle the possibility that decoding strings may return NULL and when such an error happens, properly decrement any previously decoded strings and return early. In addition, in dynload_aix, ensure that we pass the decoded string *object* pathname_ob to PyErr_SetImportError instead of the original pathname buffer. Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> (cherry picked from commit 2d2af32) Co-authored-by: Kevin Adler <kadler@us.ibm.com>
methane
approved these changes
Oct 15, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When running in a non-UTF-8 locale, if an error occurs while importing a
native Python module (say because a dependent share library is missing),
the error message string returned may contain non-ASCII code points
causing a UnicodeDecodeError.
PyUnicode_DecodeFSDefault is used for buffers which may contain
filesystem paths. For consistency with os.strerror(),
PyUnicode_DecodeLocale is used for buffers which contain system error
messages. While the shortname parameter is always encoded in ASCII
according to PEP 489, it is left decoded using PyUnicode_FromString to
minimize the changes and since it should not affect the decoding (albeit
potentially slower).
In dynload_hpux, since the error buffer contains a message generated
from a static ASCII string and the module filesystem path,
PyUnicode_DecodeFSDefault is used instead of PyUnicode_DecodeLocale as
is used elsewhere.
For both dynload_aix and dynload_hpux, properly handle the possibility
that decoding strings may return NULL and when such an error happens,
properly decrement any previously decoded strings and return early.
In addition, in dynload_aix, ensure that we pass the decoded string
object pathname_ob to PyErr_SetImportError instead of the original
pathname buffer.
Co-authored-by: Serhiy Storchaka storchaka@gmail.com
(cherry picked from commit 2d2af32)
Co-authored-by: Kevin Adler kadler@us.ibm.com
https://bugs.python.org/issue41894