New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support PEP 383 on Windows: mbcs support of surrogateescape error handler #54030
Comments
It would be nice to support PEP-383 (surrogateescape) on Windows, but the mbcs codec doesn't support it for performance reason. The Windows functions to encode/decode MBCS don't give the index of the unencodable/undecodable character/byte. For encoding, we can try to encode character by character (but be careful of surrogate pairs) and check that the character is a Python lone surrogate character or not (character in range U+DC80..U+DCFF). For decoding, it is more complex because MBCS can be a multibyte encoding, eg. cp65001 (Microsoft variant of utf-8, see bpo-6058). So it's not possible to encode byte per byte and we should write an heuristic to guess the right number of bytes for each call to the decode function. -- A completly different solution is to get the MBCS code page and use the Python code page codec (eg. "cp1252") instead of "mbcs" encoding, because Python cpXXXX codecs support all Python error handlers. Example (with Python 2.6): >>> print(u"abcŁdef".encode("cp1252", "replace"))
abc?def
>>> print(u"abcŁdef".encode("cp1252", "ignore"))
abcdef
>>> print(u"abcŁdef".encode("cp1252", "backslashreplace"))
abc\u0141def See also bpo-8611 for the problem if the Python path cannot be encoded to mbcs (work in progress, see bpo-9425). |
STINNER Victor wrote:
>
> New submission from STINNER Victor <victor.stinner@haypocalc.com>:
>
> It would be nice to support PEP 383 (surrogateescape) on Windows, but the mbcs codec doesn't support it for performance reason. The Windows functions to encode/decode MBCS don't give the index of the unencodable/undecodable character/byte. For encoding, we can try to encode character by character (but be careful of surrogate pairs) and check that the character is a Python lone surrogate character or not (character in range U+DC80..U+DCFF). For decoding, it is more complex because MBCS can be a multibyte encoding, eg. cp65001 (Microsoft variant of utf-8, see python/cpython#50308). So it's not possible to encode byte per byte and we should write an heuristic to guess the right number of bytes for each call to the decode function.
>
> --
>
> A completly different solution is to get the MBCS code page and use the Python code page codec (eg. "cp1252") instead of "mbcs" encoding, because Python cpXXXX codecs support all Python error handlers. Example (with Python 2.6):
>
>>>> print(u"abcŁdef".encode("cp1252", "replace"))
> abc?def
>>>> print(u"abcŁdef".encode("cp1252", "ignore"))
> abcdef
>>>> print(u"abcŁdef".encode("cp1252", "backslashreplace"))
> abc\u0141def That would certainly be a better approach, provided that our We could then also alias 'mbcs' to the cp-encoding (sort of |
Oh wait. PEP-383 is a solution to store undecodable bytes in an unicode string, but for mbcs I'm trying to get the opposite: store unicode in bytes and this is not possible (at least with PEP-383). Example with Python 3.1: >>> print("abcŁdef".encode("cp1252", "surrogateescape"))
...
UnicodeEncodeError: 'charmap' codec can't encode character '\u0141' in position 3: character maps to <undefined> |
Close this issue: PEP-383 is specific to filesystem using bytes, it is useless on Windows (the problem on Windows is on encoding, not on decoding). |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: