fix(chunking): normalize line endings before splitting - #2546
Merged
lyingbug merged 1 commit intoAug 5, 2026
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2535
Description
中文
修复关闭父子分块、按标题切分时,在线测试分块预览与上传文档实际分块结果不一致的问题。
浏览器文本框会将粘贴内容的 CRLF 换行符规范为 LF,而上传文档保留原始 CRLF。额外的
\r会参与字符长度和边界计算,导致相同内容在预览与正式解析中产生不同分块。本次统一在在线预览、上传文档处理和手工录入处理前,将 CRLF/CR 规范为 LF,并新增 CRLF 场景的回归测试。
使用 Issue 附件
test.md验证:关闭父子分块、按标题切分、块大小 500、重叠 20 时,预览与实际解析均生成 44 个分块。English
Fixes inconsistent chunking results between the online preview and uploaded document ingestion when parent-child chunking is disabled and heading-based splitting is used.
Browser textareas normalize pasted CRLF text to LF, while uploaded files preserve CRLF. The additional
\rcharacters affected character counts and chunk boundaries, resulting in different chunks for the same content.This change normalizes CRLF/CR line endings to LF before previewing, processing uploaded documents, and processing manually entered content. Regression tests cover CRLF input.
Verified with the issue fixture
test.md: with parent-child chunking disabled, heading splitting, chunk size 500, and overlap 20, both preview and ingestion produce 44 chunks.Type of Change
Related Issue
Fixes #2535
Testing
go test ./internal/infrastructure/chunker ./internal/handler -count=1go test ./internal/application/service -run TestBuildParentChildConfigs_PropagatesStrategy -count=1git diff --check upstream/main...HEADChecklist
git diff --check upstream/main...HEADpassesScreenshots / Recordings
Not applicable. This is a backend-only fix.