Skip to content

fix(text-chunking): split Default-mode CJK text at full-width punctuation - #441

Merged
0xShug0 merged 1 commit into
0xShug0:mainfrom
gqf2008:fix/text-chunking-cjk
Sep 4, 2026
Merged

fix(text-chunking): split Default-mode CJK text at full-width punctuation#441
0xShug0 merged 1 commit into
0xShug0:mainfrom
gqf2008:fix/text-chunking-cjk

Conversation

@gqf2008

@gqf2008 gqf2008 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

split_word_ranges in the Default text-chunking mode cut words only at ASCII spaces. CJK text has no inter-word spaces, so an entire paragraph parsed as a single word; once the text exceeded the codepoint budget (text_chunk_size, default 200) the hard-split fallback kept the whole paragraph as one chunk and the budget was never applied.

Every TTS family that goes through chunk_text_request (omnivoice, outetts, heartmula, moss_voicegen, audio8_tts, ...) is affected, in both offline and streaming runs: long Chinese text synthesized as one oversized chunk regardless of text_chunk_size. text_chunk_mode=tag_aware was the only workaround.

Fix: full-width CJK sentence/clause punctuation (。!?,、;:) now forms word boundaries — the punctuation token becomes a word of its own carrying its sentence/clause break flags, so the existing greedy budget + rollback logic splits at natural boundaries. ASCII punctuation stays attached to its word, leaving Latin text byte-identical.

Verification

New text_chunking_test (5 cases: CJK over-budget splits at sentence boundary, under-budget stays whole, clause-punctuation rollback, Latin sentence split unchanged, oversized single-word passthrough). Confirmed on macOS arm64 and across the full CI matrix (Linux cpu/vulkan, macOS, Windows):

  • 236-codepoint CJK paragraph with the default 200 budget: audio8_tts.text_chunk_count 1 → 2
  • 115-codepoint short text: output wav byte-identical to the pre-fix build
  • Latin text: chunks identical to pre-fix
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DENGINE_BUILD_TESTS=ON
cmake --build build --target text_chunking_test
./build/text_chunking_test   # text_chunking_test: all passed

…tion

split_word_ranges only cut words at ASCII spaces, so space-less CJK
paragraphs parsed as a single word and never chunked at the codepoint
budget — long Chinese text synthesized as one chunk in every TTS family
using chunk_text_request (offline and streaming alike), regardless of
text_chunk_size.

Full-width CJK sentence/clause punctuation now forms word boundaries
(the punctuation token itself carries the break flags); ASCII
punctuation stays attached to its word, leaving Latin text unchanged.
Adds text_chunking_test: CJK over/under budget, clause rollback,
Latin sentence split, oversized single-word passthrough.
@0xShug0
0xShug0 merged commit ad4bd57 into 0xShug0:main Sep 4, 2026
6 checks passed
@0xShug0

0xShug0 commented Sep 4, 2026

Copy link
Copy Markdown
Owner

@gqf2008 Merged. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants