Skip to content

fix(rerank): preserve code and math passages - #2495

Merged
lyingbug merged 1 commit into
Tencent:mainfrom
BigFishDreamWater:codex/fix-rerank-code-math
Aug 3, 2026
Merged

fix(rerank): preserve code and math passages#2495
lyingbug merged 1 commit into
Tencent:mainfrom
BigFishDreamWater:codex/fix-rerank-code-math

Conversation

@BigFishDreamWater

Copy link
Copy Markdown
Contributor

Description / 描述

English

Rerank passage cleaning removed entire fenced-code and block-LaTeX sections. Code-only or formula-only retrieval candidates therefore became empty and were skipped before reaching the rerank model.

This change removes only the Markdown delimiters while preserving the code or formula body as semantic rerank input.

中文

Rerank 文本清洗此前会删除完整的 fenced code 和块级 LaTeX 内容。仅包含代码或公式的召回候选因此会变成空字符串,并在调用 rerank 模型前被跳过。

本次修改只移除 Markdown 定界符,保留代码和公式正文作为 rerank 的语义输入。

Type of Change / 变更类型

  • Bug fix / Bug 修复
  • Test / 测试

Reproduction / 复现说明

Before this fix / 修复前:

  1. Retrieve a chunk whose content is only fenced code, for example:
```go
func answer() int { return 42 }
```
  1. Or retrieve a chunk containing only block math: $$ E = mc^2 $$.
  2. cleanPassageForRerank replaces the complete match with an empty string.
  3. getEnrichedPassage returns empty, so the candidate is excluded by the empty_passage_skip branch.

中文复现:召回仅包含 fenced code 或 $$...$$ 公式的 chunk;清洗函数删除完整匹配后返回空字符串,候选随后命中 empty_passage_skip,无法进入 rerank 模型。

Automated reproduction / 自动化复现:

go test ./internal/application/service/chat_pipeline -run TestGetEnrichedPassageKeepsCodeAndMathCandidates -count=1 -v

Before the implementation, both code_only and math_only failed with semantic-only candidate was removed from the rerank passage.

实现前,code_onlymath_only 两个子用例都会以 semantic-only candidate was removed from the rerank passage 失败。

Root Cause / 根因

The code-block and LaTeX regular expressions matched the entire block, and ReplaceAllString(..., "") deleted both formatting and semantic content.

代码块与 LaTeX 正则匹配了整个内容块,ReplaceAllString(..., "") 同时删除了格式标记和真正的语义内容。

Fix / 修复

  • Capture the inner body of fenced-code and $$...$$ blocks.

  • Replace each match with capture group $1, stripping only the delimiters.

  • Keep all other Markdown cleanup behavior unchanged.

  • Add regression coverage for code-only, math-only, mixed-code, and mixed-math passages.

  • 捕获 fenced code 与 $$...$$ 内部正文。

  • 使用捕获组 $1 替换完整匹配,只移除定界符。

  • 保持其他 Markdown 清洗行为不变。

  • 增加代码-only、公式-only、混合代码及混合公式的回归覆盖。

Testing / 测试

  • go test ./internal/application/service/chat_pipeline -count=1
  • go vet ./internal/application/service/chat_pipeline
  • git diff --check main...HEAD
  • Changed Go files formatted with gofmt / 已使用 gofmt 格式化
  • Self-reviewed the focused two-file diff / 已自审仅涉及两个文件的差异

Documentation / 文档

No user-facing API or configuration changed, so repository documentation is not affected.

未修改用户可见 API 或配置,因此无需更新仓库文档。

@lyingbug
lyingbug merged commit 60028cb into Tencent:main Aug 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants