Unicode String version 2.4.0

Module changes

Segmentation is now performed by table-driven engines generated from the state machine data published in PRI #555. The public API is unchanged — Unicode.String.split/2, next/2, stream/2, splitter/2, break?/2 and the casing functions behave exactly as before, and code using them needs no modification.

Four previously documented modules implemented the annexes by hand. They are now deprecated shims that delegate to the generated engines, so existing code keeps working and emits a compiler warning naming its replacement:

Deprecated	Replacement
Unicode.String.Break.Grapheme	Unicode.String.Dfa.Grapheme
Unicode.String.Break.Word	Unicode.String.Dfa.Word
Unicode.String.Break.Sentence	Unicode.String.Dfa.Sentence
Unicode.String.Break.Line	Unicode.String.Dfa.Line

Behaviour is unchanged and the arities match — split/1, next/1 and break?/2 for grapheme, word and line, and split/3, next/3 and break?/4 for sentence — so migrating is a module rename. The one thing to know is that the former Unicode.String.Break.Line.split/1 applied the rules alone; that is Unicode.String.Dfa.Line.rule_split/1, while Unicode.String.Dfa.Line.split/1 also runs the dictionary pass for Thai, Lao, Khmer and Burmese.

Line and sentence gain locale-aware variants (Unicode.String.Dfa.Line.split/2, Unicode.String.Dfa.Sentence.split/3) carrying the CLDR tailoring described below.

These remain internal engines rather than a supported interface; Unicode.String is the API to prefer. The shims will be removed in a future major release.

Two modules are newly public: Unicode.String.Dfa, from which the four break engines are generated, and Unicode.String.Break.Tailoring, which holds CLDR's locale tailoring and abbreviation suppressions.

Enhancements

• Support Unicode 18.0.0. Rule GB9c no longer requires a leading Indic_Conjunct_Break=Consonant, so a linker opens a conjunct sequence from any position including the start of text. Segmentation test data is refreshed to 18.0.0.

• Segment all four break types with a table-driven engine generated from the state machine data published in PRI #555. Line breaking now passes all 19,346 cases of LineBreakTest.txt where the previous engine passed 99.81%.

• Support CLDR locale tailoring of break classes through Unicode.String.Break.Tailoring. Greek sentences break at U+003B and U+037E, and ja, zh and zh-Hant line breaking treats conditional Japanese starters as ideographs rather than non-starters.

• Add an optional ICU4C backend, Unicode.String.Nif, selected with backend: :nif on Unicode.String.split/2. It is opt-in via UNICODE_STRING_NIF=true or config :unicode_string, :nif, true, requires ICU system libraries and :elixir_make, and falls back to the native implementation whenever it is unavailable, so the option is always safe to pass. See conformance.md for when it is worth enabling — end to end it is 5-7x faster for line breaking and the dictionary locales, but only 1.3-1.5x for word and grapheme breaking.

Performance

• Skip the dictionary pass in line breaking for text that cannot contain a dictionary script. Thai, Lao, Khmer and Burmese all lie in U+0E01..U+17FF, which UTF-8 encodes with a lead byte of 0xE0 or 0xE1, and neither byte can occur as a continuation byte, so a single :binary.match rules them out. Line breaking is 1.85x faster on Latin text.

• Skip the Unicode property table lookups for Latin-1 codepoints in all four break types. The break class of every codepoint below U+0100 is resolved at compile time into a tuple indexed by codepoint, and below U+00A9 no character is Extended_Pictographic or carries an Indic_Conjunct_Break value, so grapheme breaking skips those two tests entirely. Measured on 1,800 bytes of Latin text: word breaking 3.8x faster, sentence breaking 3.5x faster, line breaking 1.6x faster and grapheme breaking 1.4x faster.

• Decide grapheme and word boundaries from raw UTF-8 bytes where the answer is certain, without decoding a codepoint or consulting a property table. Two printable ASCII bytes in a row are always a grapheme boundary, and a run of ASCII letters is always a whole word provided the byte ending the run cannot join to it. Both preconditions are computed from the Unicode data at compile time. Grapheme breaking is 4.1x faster and word breaking a further 1.4x on Latin text.

• Compile the Extended_Pictographic property into a balanced binary tree of comparisons rather than a flat chain of 156 or clauses. Because or short-circuits on true, the flat form cost all 156 comparisons for every character that is not pictographic, which is almost every character in ordinary text. The tree answers in about 8 and remains valid in a guard.

Bug Fixes

• Apply a locale's casing rules to the whole string. A character with no rule for the locale switched the remainder to the locale-independent rules, so any later locale-specific mapping was lost — Lithuanian i followed by a combining dot above is the case that shows it.

• Implement the casing rules that remove a character. SpecialCasing.txt leaves a mapping blank where the character is dropped in that context, which was read as an absent mapping: the combining dot above is now removed when lower casing after a Turkish or Azeri I, and when upper or title casing after a Lithuanian soft-dotted letter.

• Apply the Lithuanian dot-above rule to J as well as I. J was excluded from the generated mappings and handled by the ASCII fast path, so it never gained the dot that an accent above requires.

• Fix Turkish and Azeri lower casing of I before a combining dot above. Before_Dot is a condition on what follows the character, but was being tested against what precedes it, so I became dotless ı in a sequence where the standard keeps the dotted i.

• Fix locale-dependent casing duplicating the start of a string. Where a contextual rule did not apply, the fallback re-cased the character with an accumulator that already held everything mapped so far, emitting that prefix twice. Affected Turkish, Azeri and Lithuanian.

• Keep dictionary-based line breaking inside its own script, so a boundary is added only between two characters of the dictionary script. Adjacent punctuation no longer becomes its own segment, which had broken after an opening bracket where LB14 forbids it and before a closing one where LB13 does.

• Apply the line-break dictionary pass in Unicode.String.stream/2 and Unicode.String.splitter/2. Both previously returned different segments from Unicode.String.split/2 for Thai, Lao, Khmer and Burmese.

• Complete LB30b with its [\p{Extended_Pictographic}&\p{Cn}] × EM alternative, so an unassigned pictographic keeps its emoji modifier. These characters carry lb=ID or lb=XX, so the rule cannot be expressed in line-break classes alone.

• Implement LB25 in full, tracking the NU (SY | IS)* number run it is defined over. Numeric prefixes and postfixes now join only where a number is actually present, so PO × OP no longer suppresses a break unless a number follows the open punctuation.

• Implement LB28a, so breaks are suppressed inside the orthographic syllables of Brahmic scripts across the AP, AK, AS, VI and VF classes and U+25CC DOTTED CIRCLE.

• Implement LB19 and LB19a, so breaks are suppressed only before a non-initial and after a non-final quotation mark, and on both sides of any quotation mark that is not surrounded by East Asian characters. Previously every quotation mark suppressed breaks on both sides unconditionally.

• Implement LB15a and LB15b, so a break is suppressed after an initial (Pi) quotation mark across any following spaces, and before a final (Pf) quotation mark that ends the text or is followed by space, glue or closing punctuation. This also removes a QU SP* × OP rule that no longer exists in UAX #14.

• Apply the LB30 East-Asian-width restriction, so (AL | HL | NU) × OP and CP × (AL | HL | NU) no longer suppress a break when the punctuation has an East_Asian_Width of F, W or H.

• Apply LB10 to a combining mark that begins a segment. A CM or ZWJ with no base to attach to is now treated as AL, where previously it kept class CM and admitted a spurious break before the following character.

• Resolve Line_Break=SA by General_Category as LB1 requires, to CM for Mn and Mc and to AL otherwise. Previously all SA resolved to AL, which broke sequences such as an ideograph followed by a Thai combining mark.