PEP 623: Remove wstr from Unicode object

[methane]

No description provided.

[vstinner]

This PEP makes me sad. I dislike having to wait until 2025 (Python 3.14) to remove the last deprecated functions :-( I would prefer to target Python 3.12 for removal, and deprecate all at once in Python 3.10.

[vstinner]

I prefer to new plan: I'm more excited to see the removal in 2023 (Python 3.12) rath in 2025 (Python 3.14)!

[vstinner]

At the beginning, the PEP was focused on the wstr member. Now it also removes PyUnicode_READY(). But the PEP title can be changed to:

"Remove the legacy Unicode C API"

[methane]

I want to focus on APIs using wstr, because most APIs using Py_UNICODE can be removed without PEP.

[vstinner]

When I read https://www.python.org/dev/peps/pep-0393/#deprecations-removals-and-incompatibilities and your PEP, it's non-obvious why functions using Py_UNICODE* like PyUnicode_Encode() are not scheduled for removal by this PEP. Even for me who knows well Unicode internals :-)

First of all, do you care of removing these functions in the "short term" (Python 3.12) or long term (later)? Or do you want to keep them for now? If you want to keep them, please just add a sentence somewhere to explain that.

For example, explain that the PEP is limited to removing PyUnicode_WCHAR_KIND and all related APIs, for reasons listed in the PEP. But functions with Py_UNICODE* don't use PyUnicode_WCHAR_KIND and so are out of the scope of this PEP.

[methane]

First of all, do you care of removing these functions in the "short term" (Python 3.12) or long term (later)? Or do you want to keep them for now? If you want to keep them, please just add a sentence somewhere to explain that.

I want to remove most of them before Python 3.12. I don't want to make Python 3.12 big breaking change.

Most APIs are documented as deprecated since Python 3.3, and marked Py_DEPRECATED since Python 3.6. Most of them can be removed in 3.10.

For example, explain that the PEP is limited to removing PyUnicode_WCHAR_KIND and all related APIs, for reasons listed in the PEP. But functions with Py_UNICODE* don't use PyUnicode_WCHAR_KIND and so are out of the scope of this PEP.

PyUnicode_AsUnicode() and "u" format in PyArg_ParseTuple are not relating to PyUnicode_WCHAR_KIND, but wstr.

[vstinner]

Most APIs are documented as deprecated since Python 3.3, and marked Py_DEPRECATED since Python 3.6. Most of them can be removed in 3.10.

I'm not sure that people are paying attention to compiler warnings emitted by Py_DEPRECATED(). Do you think that it's worth it to start emiting a DeprecationWarning in Python 3.10, and remove these functions in Python 3.12, or maybe already in Python 3.11?

[methane]

I don't think so.

For example, PyLong_FromUnicode doesn't execute arbitrary code. It doesn't release GIL too.
If we emit warning from the function, now the function will run arbitrary code (warning filter).

It is breaking change too. But compile error must be safe than runtime warning.
We can revert the removal anyway.

[vstinner]

The implementation of the Python str types and codecs take more than 21k lines of C code. I would prefer to remove as much deprecated code as possible to reduce the maintenance burden.

The problem is that I don't know how many C extension modules still use it. Maybe we can start by deprecating all of them in 3.10, announce removal in 3.12, but postpone removal of some functions using Py_UNICODE* if needed. Something like the PEP 620 process:
https://www.python.org/dev/peps/pep-0620/#process-to-reduce-the-number-of-broken-c-extensions

It sounds relevant to put it in the same PEP since it's also related to Unicode C API, but I can also understand if you prefer to keep the PEP as small as possible. If you don't want to put it in the same PEP, I'm not sure if I should write another PEP... or simply deprecate/remove with no PEP :-) A PEP helps to communicate our intent and make deprecation more visible.

$ wc -l Objects/stringlib/*.h Objects/unicodectype.c Objects/unicodeobject.c
    25 Objects/stringlib/asciilib.h
   825 Objects/stringlib/codecs.h
    27 Objects/stringlib/count.h
   116 Objects/stringlib/ctype.h
    25 Objects/stringlib/eq.h
   283 Objects/stringlib/fastsearch.h
   119 Objects/stringlib/find.h
   134 Objects/stringlib/find_max_char.h
   163 Objects/stringlib/join.h
    82 Objects/stringlib/localeutil.h
   125 Objects/stringlib/partition.h
    53 Objects/stringlib/replace.h
   390 Objects/stringlib/split.h
    27 Objects/stringlib/stringdefs.h
   740 Objects/stringlib/transmogrify.h
    25 Objects/stringlib/ucs1lib.h
    25 Objects/stringlib/ucs2lib.h
    26 Objects/stringlib/ucs4lib.h
    10 Objects/stringlib/undef.h
    31 Objects/stringlib/unicodedefs.h
  1291 Objects/stringlib/unicode_format.h
   297 Objects/unicodectype.c
 16272 Objects/unicodeobject.c
 21111 total

[vstinner]

LGTM, but I suggest to change the PEP number to 623.

You may try to merge the PR ASAP to reserve the PEP number, we can continue to discuss it after it's merged ;-)

[hugovk]

A typo and suggestion

[hugovk]

-Deprecated APIs which doesn't use the members are out of scope because
+Deprecated APIs which don't use the members are out of scope because