Create PyUnicode_EncodeFSDefault() function

[vstinner]

BPO	8715
Nosy	@malemburg, @loewis, @gpshead, @pitrou, @vstinner, @benjaminp, @ezio-melotti
Files	pyunicode_encodefsdefault-3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-05-15.16:28:41.394>
created_at = <Date 2010-05-14.16:53:52.192>
labels = ['interpreter-core', 'expert-unicode']
title = 'Create PyUnicode_EncodeFSDefault() function'
updated_at = <Date 2010-05-15.16:28:41.393>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2010-05-15.16:28:41.393>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2010-05-15.16:28:41.394>
closer = 'vstinner'
components = ['Interpreter Core', 'Unicode']
creation = <Date 2010-05-14.16:53:52.192>
creator = 'vstinner'
dependencies = []
files = ['17341']
hgrepos = []
issue_num = 8715
keywords = ['patch']
message_count = 6.0
messages = ['105721', '105722', '105779', '105809', '105810', '105819']
nosy_count = 8.0
nosy_names = ['lemburg', 'loewis', 'gregory.p.smith', 'pitrou', 'vstinner', 'benjamin.peterson', 'ezio.melotti', 'Arfrever']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue8715'
versions = ['Python 3.2']

[vstinner]

PyUnicode_EncodeFSDefault() is the opposite of PyUnicode_DecodeFSDefault(AndSize)() and is similar to the new function os.fsencode(). As you can see in the patch, it simplifies many functions.

/* Encodes a Unicode object to Py_FileSystemDefaultEncoding with the
"surrogateescape" error handler and returns a bytes object.

If Py_FileSystemDefaultEncoding is not set, fall back to UTF-8.
*/

PyAPI_FUNC(PyObject*) PyUnicode_EncodeFSDefault(
    PyObject *unicode
    );

The function unify the behaviour when Py_FileSystemDefaultEncoding is NULL: use UTF-8 whereas import uses ASCII. Other functions did already fall back to UTF-8: PyUnicode_AsEncodedString() uses PyUnicode_GetDefaultEncoding() (hardcoded to utf8 in Python3) if encoding is NULL

The patch does also fix tkinter module initializer (use surrogateescape error handler, instead of strict).

The patch was first attached to issue bpo-8611.

[vstinner]

Ooops, I attached the wrong version of the patch. Version 3 changes the documentation (Encodes => Encode).

[vstinner]

Notes for myself:

• "Encodes" and "fallback" in .h documentation => "Encode", "fall back"
• bootstrap failure on Windows: import did use default error handler, it uses surrogateescape error handler, but PyUnicode_EncodeString() doesn't have codec "fast-path" for MBCS+surrogateescape.

[vstinner]

bootstrap failure on Windows: import did use default error handler,
it uses surrogateescape error handler, but PyUnicode_EncodeString()
doesn't have codec "fast-path" for MBCS+surrogateescape.

I enabled "shortcuts" in PyUnicode_EncodeString() for any error handler (not only the default error handler, strict) in r81192.

[vstinner]

PyUnicode_AsEncodedString() contains a special path for the file system encoding. I don't think that it is still needed, but I don't know how to check that.

/* During bootstrap, we may need to find the encodings
   package, to load the file system encoding, and require the
   file system encoding in order to load the encodings
   package.

   Break out of this dependency by assuming that the path to
   the encodings module is ASCII-only.  XXX could try wcstombs
   instead, if the file system encoding is the locale's
   encoding. */
else if (Py_FileSystemDefaultEncoding &&
         strcmp(encoding, Py_FileSystemDefaultEncoding) == 0 &&
         !PyThreadState_GET()->interp->codecs_initialized)
    return PyUnicode_EncodeASCII(PyUnicode_AS_UNICODE(unicode),
                                 PyUnicode_GET_SIZE(unicode),
                                 errors);

[vstinner]

Commited as r81194 (py3k), blocked in 3.1 (r81195).