Skip to content

v5.0.0 - Rule-Based Line Breaking Architecture 🐻

Choose a tag to compare

View all tags
@SCKelemen SCKelemen released this 17 Dec 10:43
· 46 commits to main since this release
v5.0.0
c90048f

v5.0.0 - Rule-Based Line Breaking Architecture

🎯 Major Achievement: 100% UAX #14 Conformance

This release extends the rule-based state machine architecture from UAX #29 (v4.0.0) to UAX #14 (Line Breaking Algorithm), achieving 100% conformance on all 19,338 official Unicode tests.

✨ What's New

Rule-Based Line Breaking Implementation

UAX #14 now uses a clean, rule-based architecture that directly maps to the Unicode Standard specification:

  • LineBreakContext abstraction: Clean navigation API with helper methods

    • SkipBackward/SkipForward: Skip over combining marks (LB9 rule)
    • FindForward/FindBackward: Search for target classes
    • MatchSequence: Pattern matching for rule sequences

  • 59 Named rule functions: Each Unicode rule (LB4, LB5, LB8, LB21, etc.) becomes a named, testable function

  • Declarative rule chains: First-match-wins strategy with clear precedence

  • Pair table fallback: Common cases handled by efficient 2,064-entry lookup table

100% Conformance Fixes

Achieved perfect conformance by fixing these edge cases:

  1. French guillemet separators (»word« pattern)

    • Pattern: « SP ÷ AL when part of emphasis, not quotation
    • U+00AB/U+00BB require special break handling

  2. German quotes („..." and ‚...' patterns)

    • ClassQU_Pi acts as closing quote (not opening)
    • U+201E/U+201A (ClassOP) open, U+201C/U+2018 (ClassQU_Pi) close

  3. Hebrew MAQAF (U+05BE hyphen)

    • HL × HH ÷ HL pattern for Hebrew hyphen
    • New ruleLB21_HH_Break handles (HL | AL) × HH ÷ HL

  4. Regional indicators with combining marks

    • RI × CM × RI sequences
    • ruleLB30a now skips CM/ZWJ when counting RIs

  5. Extended pictographic × emoji modifier

    • Reserved emoji ranges (U+1F000-U+1FFFD)
    • ruleLB30b checks isExtendedPictographic for any base class

📊 Test Results

Total tests: 19,338
Passed: 19,338 (100.0%)
Failed: 0 (0.0%)

🏗️ Architecture Benefits

Before (Original Implementation)

  • 1,112-line monolithic function
  • Complex inline conditionals
  • Difficult to debug and extend

After (Rule-Based Implementation)

  • Isolated, independently testable rule functions
  • Direct spec mapping (ruleLB4, ruleLB21, etc.)
  • Clear documentation with spec links
  • Easy to add new rules without refactoring
  • No massive conditional chains

⚡ Performance Impact

The rule-based implementation is 2-3x slower due to abstraction overhead:

Text Length Original Rule-Based Change
Short (10 chars) 494 ns/op 1,360 ns/op 2.75x slower
Medium (64 chars) 3,934 ns/op 9,374 ns/op 2.38x slower
Long (45 chars) 2,138 ns/op 5,209 ns/op 2.44x slower

Trade-off: Performance remains excellent for text layout (thousands of characters per millisecond). The maintainability benefits far outweigh the performance cost for this use case.

🐻 License Update

Updated to BearWare 1.0 - MIT License with bear emojis:

  • Less corporate feel
  • Easy to detect in the wild
  • Shows we're weekend warriors, not a corporation

📦 New Files

  • uax14/context.go - LineBreakContext abstraction (354 lines)
  • uax14/linebreak_rules.go - Rule-based implementation (1,786 lines, 59 rule functions)
  • uax14/linebreak_rules_test.go - Test suite with conformance tests
  • uax14/LINEBREAK_RULES.md - Comprehensive rule documentation
  • LICENSE - BearWare 1.0 license with bear emoji ASCII art

🔧 Breaking Changes

None - the original implementation remains available as FindLineBreakOpportunities. The new rule-based implementation is exposed via FindLineBreakOpportunitiesWithRules for testing and comparison.

🎓 What This Means

This architecture provides:

  1. Direct spec mapping: Rule functions named after Unicode spec rules
  2. Independent testing: Each rule can be tested and traced independently
  3. Clear debugging: Rule execution can be logged to understand break decisions
  4. Easy updates: New Unicode versions can add rules without refactoring
  5. Reduced complexity: No massive conditional chains or inline state tracking

This matches the successful pattern from UAX #29 v4.0.0, providing consistency across the codebase.

🔗 References

🙏 Acknowledgments

This release demonstrates rigorous engineering while maintaining a personal, accessible approach. Made with care by weekend warriors. 🐻

Full Changelog: v4.0.0...v5.0.0

Assets 2
Loading