Merge a4e6fb1 into f057fdf

Author: raeq

CHANGELOG.md

@@ -51,6 +51,30 @@ compatibility (see [RELEASING.md](RELEASING.md)).
   key builders are byte-identical, having no confusable step on this path. Greek prose
   containing a final sigma changes, though `canonicalize` already rewrote Greek before
   this — `κόσμος` was `koouoς` and is now `koouoc`.
+- **The 54 negative enclosed letters fold to their letter (#815).** `🅐` and `🅰` folded on
+  no surface. Their positive counterparts `ⓐ` and `🄰` do, because NFKC decomposes those
+  and leaves these alone — two neighbouring blocks, opposite outcomes, and a "fancy text"
+  generator offers both side by side, so one style was neutralised and the other passed
+  through untouched.
+
+  Derived from the UCD name and filtered to what NFKC does not already handle, so the set
+  is exactly the 54 that need it: NEGATIVE CIRCLED (26), NEGATIVE SQUARED (26), CROSSED
+  NEGATIVE SQUARED and one stray SQUARED. Two families that match the same name pattern
+  are excluded because both are already handled correctly: **Tags** are stripped as a
+  smuggling class (#413), and **combining letters** are category `Mn`, which is
+  `strip_accents`' business.
+
+  Four of them are dual-purpose. `🅰` is NEGATIVE SQUARED LATIN CAPITAL LETTER A *and* the
+  blood-type A button, so it now folds under `canonicalize`, following #614 — inside a
+  comparison preset the fold wins over the name, or a spoof and its target stop being
+  equal. `llm_guardrail` still names it, because demojize runs before the fold there;
+  that divergence is #918's subject. The `build.rs` gate on the emoji/confusable overlap
+  moves 50 → 54 and caught this, which is what it exists for.
+
+  **Upgrade note — `KEY_SCHEMA_VERSION` goes 4 → 5.** `catalog_key` moves for all 54.
+  `search_key` and `sort_key` do not. The key-stability corpus contained none of these,
+  which is the **third** time this cycle the fixture stayed green through a key change
+  because its corpus did not sample the class being fixed; 58 rows added with it.
 
 - **`scripts/watch_pr.py` — a PR watcher whose decision logic is tested.** Shepherding
   loops kept being written by hand, and four bugs kept coming back:

build.rs

@@ -430,9 +430,9 @@ fn main() {
             .collect();
         assert_eq!(
             overlap.len(),
-            50,
-            "emoji_single.tsv ∩ confusables_to_latin.tsv changed: expected the 50 rows \
-             reviewed in #614 and #801, found {}. A new row means a confusable source is \
+            54,
+            "emoji_single.tsv ∩ confusables_to_latin.tsv changed: expected the 54 rows \
+             reviewed in #614, #801 and #815, found {}. A new row means a confusable source is \
              now named instead of folded inside strip_obfuscation. Review it, then update \
              this count.",
             overlap.len()

docs/architecture/data-tables.md

@@ -30,7 +30,7 @@ membership is the whole question and there is no value to store:
 | Table | Key type | Entries | Purpose |
 |---|---|---|---|
 | Hanzi pinyin | `char` | ~21K | CJK ideograph → pinyin |
-| Confusables (Latin) | `char` | 2,300 | TR39 + supplement (#342) + attested (#597) + ICANN LGR (#831) → Latin |
+| Confusables (Latin) | `char` | 2,354 | TR39 + supplement (#342) + attested (#597) + ICANN LGR (#831) → Latin |
 | Confusables (Cyrillic) | `char` | ~1,352 | TR39 confusable → Cyrillic |
 | Upstream confusable sources | `char` (set) | 6,565 | Coverage denominator (#563) |
 | Case folding | `char` | 1,557 | Unicode CaseFolding.txt |

docs/limitations.md

@@ -351,7 +351,7 @@ The [MDPI homoglyph detection paper (2022)](https://www.mdpi.com/2224-2708/11/3/
 ### The bundled table does not cover every confusable, and the gap is now measurable
 
 A TR39 fold is only as good as the table behind it, and disarm's to-Latin table is a
-*subset* of upstream `confusables.txt`: 6,565 sources upstream, 2,300 rows bundled. The
+*subset* of upstream `confusables.txt`: 6,565 sources upstream, 2,354 rows bundled. The
 gap is mostly deliberate — a source whose upstream target is CJK, Arabic or Hangul does
 not belong in a to-Latin table — but "mostly deliberate" is not a claim a deployment
 should have to take on trust.

docs/user-guide/confusables.md

@@ -351,7 +351,7 @@ By default, confusables are normalized to Latin. You can specify a different tar
 
 | Target | Mappings | Description |
 |--------|----------|-------------|
-| `"latin"` (default) | 2,300 | Non-Latin → Latin. Cyrillic а→a, Greek Ρ→P, etc. |
+| `"latin"` (default) | 2,354 | Non-Latin → Latin. Cyrillic а→a, Greek Ρ→P, etc. |
 | `"cyrillic"` | 1,352 | Non-Cyrillic → Cyrillic. Latin A→А, p→р, etc. |
 | `"arabic"` | 373 | Non-Arabic → Arabic. `⸮`→`؟`, `𞣉`→`٣`, etc. |
 | `"hebrew"` | 261 | Non-Hebrew → Hebrew. `ℵ`→`א`, `∸`→`﬩`, etc. |

python/disarm/_api.py

@@ -842,7 +842,7 @@ def normalize_confusables(
     Args:
         text: Input string potentially containing homoglyphs.
         target_script: Script to normalize toward. Supported values:
-            ``"latin"`` (default, 2,300 mappings), ``"cyrillic"`` (1,352 mappings),
+            ``"latin"`` (default, 2,354 mappings), ``"cyrillic"`` (1,352 mappings),
             ``"arabic"`` (373 mappings) and ``"hebrew"`` (261 mappings).
 
             The two RTL targets exist because generation drops an equivalence

scripts/gen_confusables.py

@@ -26,6 +26,7 @@
 from __future__ import annotations
 
 import argparse
+import re
 import sys
 import unicodedata
 import urllib.request
@@ -344,6 +345,55 @@ class through a small capital is covered without an edit here. The shape is a
     return out
 
 
+def _enclosed_letter_folds() -> dict[str, str]:
+    """`<prefix> LATIN {CAPITAL,SMALL} LETTER X` -> `x`, derived from the UCD name (#815).
+
+    U+1F150 and U+1F170 fold on no surface. Their *positive* counterparts `\u24d0` and
+    `\u1f130` fold via NFKC, which decomposes those and does not decompose these — two
+    neighbouring blocks, opposite outcomes, and nothing said so. A generator offering
+    "circled" and "circled (negative)" side by side gets one neutralised and one through
+    untouched.
+
+    Same shape as `_small_capital_folds`: the name states the letter, so there is no
+    visual judgment to make. Two families matching the pattern are deliberately excluded,
+    because both are already handled correctly and folding them would be wrong:
+
+    * **Tags** (U+E0041 and 51 more) are stripped as a smuggling class (#413), not folded.
+      `canonicalize` already returns `ab` for a tag between two letters, and
+      `has_anomalies` fires on it.
+    * **Combining letters** (`\u0363` and 22 more) are category `Mn` — diacritics written
+      above a base in medieval manuscripts, not letters standing in for one. They are
+      `strip_accents`' business.
+
+    The filter is therefore on category: a letter or a symbol, never a mark and never a
+    format character.
+    """
+    out: dict[str, str] = {}
+    for cp in range(0x110000):
+        ch = chr(cp)
+        # `.+` rather than `.*`: a bare `LATIN CAPITAL LETTER A` is ASCII `A` itself, and
+        # matching it would emit 52 identity rows.
+        match = re.fullmatch(r".+\bLATIN (CAPITAL|SMALL) LETTER ([A-Z])", unicodedata.name(ch, ""))
+        if not match or ch.isascii():
+            continue
+        if ucd_category(cp)[0] not in ("L", "S"):
+            continue
+        # The whole point of the set: NFKC already decomposes the positive forms, and
+        # a row for one of those would be redundant with a step that runs before the
+        # fold. What is left is what NFKC leaves alone — 54 code points, all of them
+        # negative, crossed or otherwise unmapped by the compatibility data.
+        if ucd_nfkc(cp) != ch:
+            continue
+        # Case comes from the NAME, not from `fix_case_mismatch`. These are category `So`,
+        # so the case reconciler cannot tell a capital from a small letter and left every
+        # row lowercase — which made U+1F170 fold to `a` while its positive counterpart
+        # U+1F130 reaches `A` through NFKC. Two spellings of the same style disagreeing is
+        # the asymmetry this set exists to remove, so it must not be reintroduced here.
+        letter = match.group(2)
+        out[ch] = letter if match.group(1) == "CAPITAL" else letter.lower()
+    return out
+
+
 def _close_under_case(fold: dict[str, str]) -> dict[str, str]:
     """Give every entry's case pair the same ASCII letter (#801).
 
@@ -365,7 +415,7 @@ def _close_under_case(fold: dict[str, str]) -> dict[str, str]:
     return out
 
 
-ASCII_FOLD = _close_under_case({**_small_capital_folds(), **ASCII_FOLD})
+ASCII_FOLD = _close_under_case({**_enclosed_letter_folds(), **_small_capital_folds(), **ASCII_FOLD})
 
 
 # ---------------------------------------------------------------------------
@@ -959,8 +1009,14 @@ def generate_mappings(
         # policy question in #815, not this.
         #
         # An existing row always wins: this only fills gaps.
-        for glyph, letter in _small_capital_folds().items():
-            merged.setdefault(ord(glyph), fix_case_mismatch(ord(glyph), letter))
+        #
+        # `_enclosed_letter_folds` joins it for the same reason and on the same terms
+        # (#815). U+1F150 and U+1F170 fold on no surface while their positive
+        # counterparts fold via NFKC, so a generator offering "circled" and "circled
+        # (negative)" side by side gets one neutralised and one through untouched.
+        for source in (_small_capital_folds(), _enclosed_letter_folds()):
+            for glyph, letter in source.items():
+                merged.setdefault(ord(glyph), fix_case_mismatch(ord(glyph), letter))
         # #342/#343: measured cross-script supplement, applied with priority so it
         # can add a missing fold or re-point an existing one.
         for cp, target in (supplement or {}).items():

src/api/metadata.rs

@@ -164,7 +164,15 @@ pub fn unicode_version() -> &'static str {
 /// `\u0237` ȷ to `j`, `\u03a3` Σ to `С`, `\u2502` │ to `ӏ`. That moves `canonicalize`,
 /// `canonicalize_strict`, `strip_obfuscation` and `normalize_confusables`; the three key
 /// builders do not fold confusables through this path and are byte-identical.
-pub const KEY_SCHEMA_VERSION: u32 = 5;
+///
+/// Bumped to 6 by #815, for the negative enclosed letters. U+1F150 and U+1F170 fold
+/// on no surface while their positive counterparts fold via NFKC, so a generator offering
+/// "circled" and "circled (negative)" side by side got one neutralised and one through
+/// untouched. 54 rows, and `catalog_key` moves for all of them. The corpus held none of
+/// these either — third time in this cycle that the fixture stayed green through a key
+/// change because its corpus did not sample the class being fixed, so 58 rows were added
+/// with it.
+pub const KEY_SCHEMA_VERSION: u32 = 6;
 
 /// SHA-256 of the key-stability fixture's *decompressed* bytes (#887).
 ///
@@ -199,7 +207,7 @@ pub const KEY_SCHEMA_VERSION: u32 = 5;
 /// difference was `# generated against disarm 0.14.1` becoming `0.15.0`. The rows are
 /// the semantic anchor: they change when, and only when, a key moved.
 pub const KEY_FIXTURE_SHA256: &str =
-    "e57cb67ff92953a3fe31e41522436d59b8d133651f8fa51c2841a6f619dae5b6";
+    "6a1c8a2781e586210088c7322c4ad18248c414185815a9531859e94ee9f40435";
 
 /// The key-schema counter, as a function (#645).
 ///

src/tables/confusables_data.rs

@@ -4,7 +4,7 @@
 //! release is not restated here — [`CONFUSABLES_VERSION`] carries it, parsed from the
 //! TSV header by build.rs so there is exactly one place it can be wrong (#560).
 //!
-//! Contains 2,300 non-Latin → Latin mappings and 1,352 non-Cyrillic →
+//! Contains 2,354 non-Latin → Latin mappings and 1,352 non-Cyrillic →
 //! Cyrillic mappings. Uses compile-time perfect hash maps (`phf`) for O(1)
 //! lookups. Covers Cyrillic, Greek, Armenian, Georgian, CJK compatibility,
 //! mathematical symbols, fullwidth forms, and other confusable characters.