fix(rerank): preserve code and math passages - #2495
Merged
lyingbug merged 1 commit intoAug 3, 2026
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description / 描述
English
Rerank passage cleaning removed entire fenced-code and block-LaTeX sections. Code-only or formula-only retrieval candidates therefore became empty and were skipped before reaching the rerank model.
This change removes only the Markdown delimiters while preserving the code or formula body as semantic rerank input.
中文
Rerank 文本清洗此前会删除完整的 fenced code 和块级 LaTeX 内容。仅包含代码或公式的召回候选因此会变成空字符串,并在调用 rerank 模型前被跳过。
本次修改只移除 Markdown 定界符,保留代码和公式正文作为 rerank 的语义输入。
Type of Change / 变更类型
Reproduction / 复现说明
Before this fix / 修复前:
$$ E = mc^2 $$.cleanPassageForRerankreplaces the complete match with an empty string.getEnrichedPassagereturns empty, so the candidate is excluded by theempty_passage_skipbranch.中文复现:召回仅包含 fenced code 或
$$...$$公式的 chunk;清洗函数删除完整匹配后返回空字符串,候选随后命中empty_passage_skip,无法进入 rerank 模型。Automated reproduction / 自动化复现:
go test ./internal/application/service/chat_pipeline -run TestGetEnrichedPassageKeepsCodeAndMathCandidates -count=1 -vBefore the implementation, both
code_onlyandmath_onlyfailed withsemantic-only candidate was removed from the rerank passage.实现前,
code_only与math_only两个子用例都会以semantic-only candidate was removed from the rerank passage失败。Root Cause / 根因
The code-block and LaTeX regular expressions matched the entire block, and
ReplaceAllString(..., "")deleted both formatting and semantic content.代码块与 LaTeX 正则匹配了整个内容块,
ReplaceAllString(..., "")同时删除了格式标记和真正的语义内容。Fix / 修复
Capture the inner body of fenced-code and
$$...$$blocks.Replace each match with capture group
$1, stripping only the delimiters.Keep all other Markdown cleanup behavior unchanged.
Add regression coverage for code-only, math-only, mixed-code, and mixed-math passages.
捕获 fenced code 与
$$...$$内部正文。使用捕获组
$1替换完整匹配,只移除定界符。保持其他 Markdown 清洗行为不变。
增加代码-only、公式-only、混合代码及混合公式的回归覆盖。
Testing / 测试
go test ./internal/application/service/chat_pipeline -count=1go vet ./internal/application/service/chat_pipelinegit diff --check main...HEADgofmt/ 已使用gofmt格式化Documentation / 文档
No user-facing API or configuration changed, so repository documentation is not affected.
未修改用户可见 API 或配置,因此无需更新仓库文档。