✨ feat(detect): add Unicode normalization (UAX #15)

[gaborbernat]

turbohtml.detect.normalize(form, text) and turbohtml.detect.is_normalized(form, text) add Unicode normalization (UAX #15) in C, for all four forms: NFC, NFD, NFKC, and NFKD. They replace unicodedata.normalize and unicodedata.is_normalized.

How it works

The engine decomposes each code point over pinned tables (canonical mappings for the C and D forms, compatibility mappings for the K forms), puts the combining marks in canonical order, then recomposes for the composing forms. A UAX #15 quick check returns already-normalized text as the same object, so the common case skips allocation. Hangul (de)composes by arithmetic, with no table rows.

tools/generate_normalize.py builds the combining-class, decomposition, and composition tables from the interpreter's own unicodedata, gated on Unicode 16.0.0, so the C engine reimplements the data unicodedata carries and matches it. The NFC/NFKC quick-check property comes from the SHA-pinned DerivedNormalizationProps.txt. The module is self-contained and shares no table with url/idna.c, whose private NFC stays bound to UTS #46.

Competitor study

I studied lexbor and utf8proc. From utf8proc I took the decompose, canonical-order, recompose pipeline and the recomposition rule: a canonical length-2 decomposition whose first element is a starter and whose result is not a composition exclusion. From lexbor I took the quick-check fast path, where a code point marked No or Maybe, or a combining mark out of canonical order, drops to the full pipeline and everything else short-circuits. Streaming stays out of scope, which leaves the fast whole-string implementation as the floor.

Also

• A tools/bench normalize op measured against stdlib unicodedata.normalize, plus a unicodedata competitor module.
• A mapping in the stdlib migration guide, a how-to snippet, and the reference autofunctions.

Closes #543

[codspeed-hq]

Merging this PR will not alter performance

✅ 62 untouched benchmarks
🆕 1 new benchmark
⏩ 18 skipped benchmarks¹

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
🆕	test_feature[normalize]	N/A	5 ms	N/A

---

_{Comparing gaborbernat:feat/543-unicode-normalize (4ff37e8) with main (053eeb2)}

Footnotes

1. 18 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩