Skip to content

Unicode version 2.1.0

Choose a tag to compare

@kipcole9 kipcole9 released this 18 Sep 07:34
· 9 commits to main since this release

Enhancements

  • Resolves the property value that a UCD @missing annotation declares as the default. Grapheme_Cluster_Break, Word_Break and Sentence_Break now answer Other (and its XX alias), and Line_Break answers XX, for every codepoint the data files do not list. Line_Break previously covered only 358,578 of the 1,114,112 codepoints; the remainder, including all private use, resolved only through a lookup fallback and was invisible to anything enumerating the property. Where a value is both listed explicitly and named as the @missing default, as XX is for Line_Break, the two sets are merged rather than replaced.

  • Adds the age property value aliases from PropertyValueAliases.txt, so Unicode.Age.fetch("V18_0") resolves the same set as Unicode.Age.fetch("18.0").

  • Updates the underlying data to the final Unicode 18.0 release (16 September 2026), adding the Jurchen, Proto_Cuneiform and Seal scripts, seven new blocks and ten new Crown_* joining groups.

  • Adds Unicode.ScriptExtensions for the Script_Extensions (scx) property, the last UCD enumerated property without a backing module and a UTS #18 RL1.2 conformance requirement. Unicode.ScriptExtensions.script_extensions/1 returns the set of scripts a codepoint is used with, defaulting to its Script value where the UCD lists no explicit set.

  • Unicode.CharacterName.to_codepoint/2 accepts a :fuzzy option that resolves a misspelled name by String.jaro_distance/2, either as true for the default distance of 0.8 or as an explicit distance. It succeeds only where one name is strictly closest than all others, so an ambiguous query returns :error rather than guessing.

  • Adds Unicode.CharacterName.to_name/1, the reverse of to_codepoint/1, returning the Name property of a codepoint. Each name is still stored only once — the original is reconstructed from the normalized form plus separator positions — so reverse lookup adds about 410KB rather than a second name table, and leaves to_codepoint/1 unchanged in speed.

  • Unicode.CharacterName.to_codepoint/1 now resolves algorithmically derived names — CJK and Tangut ideographs, Hangul syllables, and the Seal and Jurchen characters — covering a further 131,576 characters. The names are computed from the UAX #44 derivation rules on a table-lookup miss rather than stored, adding under 4KB.

  • mix unicode.download now resolves the UCD and emoji trees through independent release channels, settable per run with --release / --emoji-release, by environment variable, or in application config. Adds --into to download to a scratch directory, --dry-run, and a post-download check that every data file reports the same Unicode version.

  • Adds the enumerated property modules Unicode.Age, Unicode.NumericType, Unicode.NumericValue, Unicode.DecompositionType, Unicode.HangulSyllableType, Unicode.IndicPositionalCategory, Unicode.VerticalOrientation, Unicode.JoiningGroup and Unicode.BidiPairedBracketType, each with a codepoint lookup, range introspection and a top-level delegate on Unicode.

  • Adds the normalization quick check modules Unicode.NfcQuickCheck, Unicode.NfdQuickCheck, Unicode.NfkcQuickCheck and Unicode.NfkdQuickCheck (the NFC_QC, NFD_QC, NFKC_QC and NFKD_QC properties) with Unicode.nfc_quick_check/1 and friends.

  • Adds Bidi_Mirrored as a first class boolean property, so Unicode.Property.bidi_mirrored?/1 and Unicode.properties/1 now report it.

  • The Name property now resolves through Unicode.fetch_property/1 to Unicode.CharacterName.

  • Adds the UTS #58 link properties Unicode.LinkTerm, Unicode.LinkEmail and Unicode.LinkBracket, which together drive link detection in flowing text. mix unicode.download gains a linkification tree alongside the UCD and emoji trees to fetch them.

  • Adds the LC (Cased_Letter) general category, the union of Lu, Ll and Lt. It is the only General_Category group whose name is not a single letter, so it was not produced by the derivation that yields L, N, P and the rest, and Unicode.GeneralCategory.fetch("lc") returned :error despite the alias being advertised.

Bug fixes

  • Property aliases containing an underscore (for example nfc_qc) are now reachable through Unicode.fetch_property/1, which previously only matched the whitespace-stripped canonical form.

  • Scripts and blocks whose PropertyValueAliases line carries more than two names now resolve by every spelling. Unicode.Script.fetch/1 failed for "copt" and "qaac", and Unicode.Block.fetch/1 for "latin1sup" and "cyrillicsupplementary", because the alias map was built by inverting a many-to-one map and elected a name absent from the data.

  • Unicode.version/0 no longer reports a stale version after the data files are updated. It derives the version from blocks.txt at compile time but did not declare it as an @external_resource, so the module was not recompiled when the data changed.

  • mix unicode.download no longer requests a non-existent emoji URL. The emoji files are versioned separately from the UCD and Public/emoji/17.0/ was never published, so the previous version-interpolated path returned 404 and the files had to be updated by hand.