Skip to content

v15.1.11 - UTS #46 preprocessing

Choose a tag to compare

View all tags
@SorinGFS SorinGFS released this 16 Aug 22:05
· 12 commits to master since this release
v15.1.11
d2958db

UTS #46 preprocessing

Summary

This update separates UTS #46 preprocessing from final IDNA eligibility.

A source code point may be permitted during nontransitional preprocessing but still be invalid after mapping and NFC. Conversely, a preprocessing sequence such as Hangul Jamo may normalize into a valid final syllable. The validator now represents and checks those phases independently without adding another label scan.

The update also corrects ACE handling and the RFC 5892 ZWNJ joining-context check.

What is new

Two-phase IDNA processing

  • UTS #46 preprocessing uses sparse uts46_ranges overrides and falls back to ranges.
  • Final post-NFC validation uses only ranges.
  • Every non-empty UTS #46 preprocessing mapping is available, even when its result must later be rejected.
  • Final eligibility is checked in the existing per-code-point contextual loop, preserving one label code-point spread and one loop.
  • Processing remains nontransitional, with STD3 rules disabled.

This resolves cases that one combined property could not represent:

  • U+00BC VULGAR FRACTION ONE QUARTER is allowed to map, but its result contains ineligible U+2044 FRACTION SLASH and is rejected afterward.
  • Hangul Jamo are permitted during preprocessing when NFC can compose them into an eligible Hangul syllable.
  • U+0133 LATIN SMALL LIGATURE IJ maps to ij.
  • U+180C MONGOLIAN FREE VARIATION SELECTOR THREE is removed, so a᠌b validates as ab.

ACE and normalization

  • Ordinary input is mapped and normalized before ACE-prefix recognition.
  • Mixed-case ACE labels are canonicalized by preprocessing before the existing strict round-trip comparison.
  • An xn-- prefix created by mapping is recognized and validated as ACE.
  • Decoded Punycode is validated without remapping or normalization.
  • A decoded A-label that is not already NFC is rejected. For example, xn--u-ccb no longer passes by being normalized into validity.
  • The existing raw apparent-A-label/non-ASCII error retains its precedence.

Final eligibility

Mapped/NFC output and decoded ACE output are checked against the final/default ranges. Only valid and nontransitional deviation properties are accepted at this stage.

The final check reuses the existing IdnaUnicodeError and runs inside the existing contextual loop. Length, hyphen, and leading-mark checks retain their current ordering.

ZWNJ joining context

The RFC 5892 Appendix A.1 check now correctly requires:

  • L or D as the nearest non-transparent joining type on the left; and
  • R or D as the nearest non-transparent joining type on the right.

A ZWNJ immediately following a virama remains accepted. The correction preserves the existing joining-type accumulator, loop, exception type, and error text.

The compact condition was compared with the RFC rule across 597,870 synthetic joining sequences, including transparent characters and both joining types found among Unicode 15.1 viramas, with no mismatch.

Unicode data

The compact table is generated directly from authoritative Unicode 15.1.0 text files:

  • IdnaMappingTable.txt;
  • DerivedCombiningClass.txt;
  • DerivedJoiningType.txt;
  • DerivedBidiClass.txt.

IdnaTestV2.txt provides the official qualification vectors.

The validator continues to ship one immutable table and does not select a Unicode version at runtime. Unicode text files remain the source of truth; generation and exhaustive source-fidelity checks stay outside the validator package.

Current compact data totals:

Data Count
Final/default ranges 2,034
Preprocessing override ranges 452
Code points covered by preprocessing overrides 9,490
Preprocessing mappings 6,239
Viramas 65
Bidi ranges 1,530
Joining-type ranges 506

The preprocessing overrides cover 8,900 valid and 590 mapped code points.

Qualification

The exact candidate implementation and Unicode 15.1 table pass:

Validator test suite          102 / 102
Applicable IdnaTestV2 cases  6077 / 6077

The 102-case validator suite includes focused coverage for:

  1. U+00A0 mapping to invalid U+0020 SPACE;
  2. U+00BC mapping through invalid U+2044 FRACTION SLASH;
  3. mapped-range interior U+0133;
  4. ignored-range interior U+180C.

The qualification adapter separately records:

  • 137 successful pure-UTS NV8/XV8 cases outside the validator's final policy;
  • 37 successful trailing-root cases outside the validator's intentional empty-final-label policy.

The runtime versions record test provenance; classification remains based on the bundled Unicode 15.1 table.

The generated data also passes exhaustive source-fidelity, range coverage, ordering, overlap, scalar, derived-property, and deterministic-rebuild checks.

Release notes

  • Correct mapped and ignored range-interior handling.
  • Permit preprocessing sequences that NFC transforms into eligible output.
  • Reject mapped output that remains IDNA-ineligible after NFC.
  • Accept mixed-case and mapping-created ACE prefixes after preprocessing.
  • Reject decoded A-labels that are not already NFC.
  • Correct invalid ZWNJ joining-context acceptance.
  • Preserve existing APIs, error behavior, nontransitional processing, disabled STD3 preprocessing, and trailing-root policy.
Assets 2
Loading