fix(text-chunking): split Default-mode CJK text at full-width punctuation - #441
Merged
Merged
Conversation
…tion split_word_ranges only cut words at ASCII spaces, so space-less CJK paragraphs parsed as a single word and never chunked at the codepoint budget — long Chinese text synthesized as one chunk in every TTS family using chunk_text_request (offline and streaming alike), regardless of text_chunk_size. Full-width CJK sentence/clause punctuation now forms word boundaries (the punctuation token itself carries the break flags); ASCII punctuation stays attached to its word, leaving Latin text unchanged. Adds text_chunking_test: CJK over/under budget, clause rollback, Latin sentence split, oversized single-word passthrough.
Owner
|
@gqf2008 Merged. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
split_word_rangesin the Default text-chunking mode cut words only at ASCII spaces. CJK text has no inter-word spaces, so an entire paragraph parsed as a single word; once the text exceeded the codepoint budget (text_chunk_size, default 200) the hard-split fallback kept the whole paragraph as one chunk and the budget was never applied.Every TTS family that goes through
chunk_text_request(omnivoice, outetts, heartmula, moss_voicegen, audio8_tts, ...) is affected, in both offline and streaming runs: long Chinese text synthesized as one oversized chunk regardless oftext_chunk_size.text_chunk_mode=tag_awarewas the only workaround.Fix: full-width CJK sentence/clause punctuation (。!?,、;:) now forms word boundaries — the punctuation token becomes a word of its own carrying its sentence/clause break flags, so the existing greedy budget + rollback logic splits at natural boundaries. ASCII punctuation stays attached to its word, leaving Latin text byte-identical.
Verification
New
text_chunking_test(5 cases: CJK over-budget splits at sentence boundary, under-budget stays whole, clause-punctuation rollback, Latin sentence split unchanged, oversized single-word passthrough). Confirmed on macOS arm64 and across the full CI matrix (Linux cpu/vulkan, macOS, Windows):audio8_tts.text_chunk_count1 → 2