Skip to content

fix(chunking): normalize line endings before splitting - #2546

Merged
lyingbug merged 1 commit into
Tencent:mainfrom
BigFishDreamWater:codex/fix-chunk-preview-line-endings
Aug 5, 2026
Merged

fix(chunking): normalize line endings before splitting#2546
lyingbug merged 1 commit into
Tencent:mainfrom
BigFishDreamWater:codex/fix-chunk-preview-line-endings

Conversation

@BigFishDreamWater

@BigFishDreamWater BigFishDreamWater commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Fixes #2535

Description

中文

修复关闭父子分块、按标题切分时,在线测试分块预览与上传文档实际分块结果不一致的问题。

浏览器文本框会将粘贴内容的 CRLF 换行符规范为 LF,而上传文档保留原始 CRLF。额外的 \r 会参与字符长度和边界计算,导致相同内容在预览与正式解析中产生不同分块。

本次统一在在线预览、上传文档处理和手工录入处理前,将 CRLF/CR 规范为 LF,并新增 CRLF 场景的回归测试。

使用 Issue 附件 test.md 验证:关闭父子分块、按标题切分、块大小 500、重叠 20 时,预览与实际解析均生成 44 个分块。

English

Fixes inconsistent chunking results between the online preview and uploaded document ingestion when parent-child chunking is disabled and heading-based splitting is used.

Browser textareas normalize pasted CRLF text to LF, while uploaded files preserve CRLF. The additional \r characters affected character counts and chunk boundaries, resulting in different chunks for the same content.

This change normalizes CRLF/CR line endings to LF before previewing, processing uploaded documents, and processing manually entered content. Regression tests cover CRLF input.

Verified with the issue fixture test.md: with parent-child chunking disabled, heading splitting, chunk size 500, and overlap 20, both preview and ingestion produce 44 chunks.

Type of Change

  • 🐛 Bug fix
  • ✨ New feature
  • 💥 Breaking change
  • 📚 Documentation update
  • 🎨 Refactor
  • ⚡ Performance improvement
  • 🧪 Test
  • 🔧 Configuration / Build / CI

Related Issue

Fixes #2535

Testing

  • go test ./internal/infrastructure/chunker ./internal/handler -count=1
  • go test ./internal/application/service -run TestBuildParentChildConfigs_PropagatesStrategy -count=1
  • Used the downloaded issue fixture to verify that normalized upload content and LF preview content both produce 44 chunks.
  • git diff --check upstream/main...HEAD

Checklist

  • git diff --check upstream/main...HEAD passes
  • Changed source files are formatted
  • Targeted tests for the changed packages/components pass
  • Diff-scoped lint was not run
  • Full-repository checks were not run; full service tests are affected by existing Windows SQLite temporary-file lock failures in unrelated data-source deletion tests
  • Self-reviewed the code
  • Added/updated tests covering the change
  • Documentation update is not required for this internal behavior fix
  • No breaking changes

Screenshots / Recordings

Not applicable. This is a backend-only fix.

@lyingbug
lyingbug merged commit 520be78 into Tencent:main Aug 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: 知识库设置->分块设置->测试分块效果 与实际分块效果不一致

2 participants