Skip to content

v1.3.0 - UAX #29 Grapheme Cluster Support

Choose a tag to compare

@deths74r deths74r released this 27 Dec 06:29
· 73 commits to main since this release

What's New

UAX #29 Grapheme Cluster Segmentation

This release adds full Unicode grapheme cluster support, enabling proper cursor movement and text segmentation for:

  • Emoji sequences: πŸ‘¨β€πŸ‘©β€πŸ‘§ (family), πŸ‘‹πŸ» (skin tones), πŸ³οΈβ€πŸŒˆ (ZWJ sequences)
  • Combining marks: Γ© (e + acute), Γ₯Μ„ (a + ring + macron)
  • Flag sequences: πŸ‡¨πŸ‡¦ πŸ‡―πŸ‡΅ (regional indicators)
  • Hangul: κ°€ (precomposed) and 각 (jamo sequences)
  • Indic scripts: Hindi/Devanagari conjuncts like ΰ€•ΰ₯ΰ€·

Highlights

  • βœ… 100% pass rate on official Unicode GraphemeBreakTest.txt (766 test cases)
  • Implements all break rules: GB3-GB13, GB999
  • Includes GB9c for Indic conjunct sequences (new in Unicode 15.1)
  • Library size: ~18KB compiled (still lightweight!)

API

// Navigate by grapheme cluster (what users perceive as "characters")
int utflite_next_grapheme(const char *text, int length, int offset);
int utflite_prev_grapheme(const char *text, int offset);

Data Tables Added

  • GCB (Grapheme Cluster Break) properties: 557 ranges (~6.7KB)
  • InCB (Indic Conjunct Break) Linkers: 43 codepoints
  • InCB Consonants: 145 ranges (~1.2KB)

Full Changelog: 92d6485...v1.3.0