Skip to content

fix(chunks): 恢复未编辑分块的按位置合并 - #2555

Merged
lyingbug merged 3 commits into
Tencent:mainfrom
mdrkrg:fix/chunk-edit-partial-downgrade
Aug 6, 2026
Merged

fix(chunks): 恢复未编辑分块的按位置合并#2555
lyingbug merged 3 commits into
Tencent:mainfrom
mdrkrg:fix/chunk-edit-partial-downgrade

Conversation

@mdrkrg

@mdrkrg mdrkrg commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Description

背景:为了支持“编辑分块”,上游把检索结果的合并逻辑从“按位置”改成了“按文本匹配”,本意是防编辑后位置失效,代价是没编辑过的文档也一起降级。因此会存在三种问题:

  • 大 overlap 分块内容重复(稳定触发):文本匹配的重叠识别窗口上限固定 400 个字符,当相邻分块的真实重叠超过 400 时,整段重叠原样重复进模型上下文
  • 被裁剪内容留下的真实间隙被按 ChunkIndex 强行拼接,语义上不相邻的两段会被用伪造的 \n\n 连在一起
  • 表格/日志等短周期文本偶发误判包含,丢失内容

改动:只对位置可信的分块恢复按位置合并,其余情况保持现有文本匹配。

  • types.SearchResult 新增内部字段 ContentRevision
  • 新增可信判定:分块没被编辑过 + 区间有效 + 内容长度与区间一致(runeLen(Content) == EndAt-StartAt),若满足则按位置合并
  • 按位置合并的三种情况:
    • 两个块有间隙则不合并
    • 有重叠则裁剪重叠部分,先按位置估计重叠量,再用文本匹配裁剪
    • 一个块完全包含另一个,确认文本相同才合并

Type of Change

  • 🐛 Bug fix
  • ✨ New feature
  • 💥 Breaking change
  • 📚 Documentation update
  • 🎨 Refactor
  • ⚡ Performance improvement
  • 🧪 Test
  • 🔧 Configuration / Build / CI

Related Issue

Testing

新增 merge_position_path_test.go

Checklist

  • git diff --check origin/main...HEAD passes
  • Changed source files are formatted
  • Targeted tests for the changed packages/components pass
  • Diff-scoped lint passes where applicable (for Go: golangci-lint run --new-from-rev=origin/main ./...)
  • Full-repository checks were run, or any unrelated/environment-dependent failures are documented above
  • Self-reviewed the code
  • Added/updated tests covering the change
  • Updated related documentation (README, docs/, Swagger annotations, etc.)
  • Breaking changes are clearly called out in the description above

Screenshots / Recordings

mdrkrg added 3 commits August 6, 2026 11:02
- Feature contracts: trusted pairs trim overlaps, join adjacent chunks
  seamlessly, and stay separate across position gaps.
- Safety contracts: never drop current content on coordinates alone.
Chunk editing made parser coordinates unreliable, so all merges were
downgraded to pure text matching, including unedited documents.

Restore position-aware merging for pairs whose coordinates are still
trustworthy, and keep the text fallback for edited or stale content so
it is never dropped on coordinates.
@lyingbug
lyingbug merged commit b7b8562 into Tencent:main Aug 6, 2026
2 checks passed
@mdrkrg

mdrkrg commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

稍等,发现个问题……

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants