Speed up unicode normalization of ASCII strings

[stevendaprano]

BPO	44987
Nosy	@vstinner, @ezio-melotti, @stevendaprano, @serhiy-storchaka, @corona10
PRs	bpo-44987: Speed up unicode normalization of ASCII strings #28283 bpo-44987: Fix typo whatsnew 3.11 #28293

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-09-11.15:05:19.089>
created_at = <Date 2021-08-24.02:07:52.797>
labels = ['type-feature', 'expert-unicode', '3.11']
title = 'Speed up unicode normalization of ASCII strings'
updated_at = <Date 2021-09-11.19:06:01.839>
user = 'https://github.com/stevendaprano'

bugs.python.org fields:

activity = <Date 2021-09-11.19:06:01.839>
actor = 'corona10'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-11.15:05:19.089>
closer = 'serhiy.storchaka'
components = ['Unicode']
creation = <Date 2021-08-24.02:07:52.797>
creator = 'steven.daprano'
dependencies = []
files = []
hgrepos = []
issue_num = 44987
keywords = ['patch']
message_count = 5.0
messages = ['400192', '401342', '401639', '401641', '401646']
nosy_count = 5.0
nosy_names = ['vstinner', 'ezio.melotti', 'steven.daprano', 'serhiy.storchaka', 'corona10']
pr_nums = ['28283', '28293']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue44987'
versions = ['Python 3.11']

[stevendaprano]

I think there is an opportunity to speed up some unicode normalisations significantly.

In 3.9 at least, the normalisation appears to be dependent on the length of the string:

    >>> setup="from unicodedata import normalize; s = 'reverse'"
    >>> t1 = Timer('normalize("NFKC", s)', setup=setup)
    >>> setup="from unicodedata import normalize; s = 'reverse'*1000"
    >>> t2 = Timer('normalize("NFKC", s)', setup=setup)
    >>> 
    >>> min(t1.repeat(repeat=7))
    0.04854234401136637
    >>> min(t2.repeat(repeat=7))
    9.98313440399943

But ASCII strings are always in normalised form, for all four normalisation forms. In CPython, with PEP-393 (Flexible String Representation), it should be a constant-time operation to detect whether a string is pure ASCII, and avoid scanning the string or attempting the normalisation.

[vstinner]

Well, someone should write a PR for it.

[corona10]

Well, someone should write a PR for it.

Well, I sent a patch :)

[serhiy-storchaka]

New changeset 9abd07e by Dong-hee Na in branch 'main':
bpo-44987: Speed up unicode normalization of ASCII strings (GH-28283)
9abd07e

[corona10]

New changeset 5277ffe by Dong-hee Na in branch 'main':
bpo-44987: Fix typo whatsnew 3.11 (GH-28293)
5277ffe