Skip to content

core-v0.0.8

Choose a tag to compare

View all tags
@chonknick chonknick released this 12 Mar 23:21
core-v0.0.8
af453ff
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

@chonkiejs/core v0.0.8

Bug Fixes

  • Fix Tokenizer encode/decode for non-BMP Unicode characters (emoji, etc.) — was using charCodeAt/fromCharCode, now uses codePointAt/fromCodePoint (#33)
  • Fix RecursiveChunker sentence-level splitting being completely non-functional — multi-char delimiters like ['. ', '! ', '? '] were joined into a string where space became a delimiter, causing all segments to collapse into one (#39)
  • Fix TokenChunker silent index failure — replaced fragile indexOf-based lookup with deterministic offset tracking (#35)
  • Fix TableChunker startIndex/endIndex inconsistency between single and multi-chunk cases (#36)
  • Consolidate duplicate WASM initialization from FastChunker and RecursiveChunker into shared module (#37)

Tests

  • Add Tokenizer unit tests (9 tests) — encode/decode round-trip, Unicode, create factory
  • Add TokenChunker test coverage (14 tests) — chunking, overlap, indices, Unicode (#34)
  • Add TableChunker index consistency test
  • Total tests: 47 → 71
Assets 2
Loading