Skip to content

fix: remove unsupported "thinking" param from evaluation judge request to avoid 400 error on non-thinking models - #134

Merged
apapi merged 3 commits into
OpenCSGs:mainfrom
apapi:fix_eval_judge_model_param
Sep 2, 2026
Merged

fix: remove unsupported "thinking" param from evaluation judge request to avoid 400 error on non-thinking models#134
apapi merged 3 commits into
OpenCSGs:mainfrom
apapi:fix_eval_judge_model_param

Conversation

@apapi

@apapi apapi commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

fix: remove unsupported "thinking" param from evaluation judge request

The evaluationChatCompletion function hard-coded thinking: {"type": "disabled"} in the judge request body, but not all upstream models support this parameter. Models like 'GPT-4.1 mini' reject it with "400: Unrecognized request argument supplied: thinking", causing the entire Listwise evaluation job to fail.
The thinking-disable intent is already handled downstream by the engine layer: sanitizeOpenAIRequestBody() applies enable_thinking: false or thinking: {type: "disabled"} only for models that are known to support it (Qwen3, QwQ, GLM, etc.). Removing the unconditional hard-code lets the engine decide per-model, fixing the 400 error on non-thinking models while preserving the disable behavior where supported.

Fix #131

@ganisback

Copy link
Copy Markdown
Collaborator

PR #134 有没有问题:有问题,不建议按当前写法合

修复方向对:GPT-4.1 mini 这类严格 OpenAI 兼容上游收到 thinking 参数就 400(issue #131),删掉硬编码可以修好它。但 PR
的论证与实际代码不符,且会引入回归:

问题 1 — 论证与代码不符。 PR body 声称"引擎层 sanitizeOpenAIRequestBody() 会对 Qwen3/QwQ/GLM 等按模型注入
enable_thinking: false 或 thinking: {type: disabled}"。实际不是这样:

• sanitizeOpenAIRequestBody(internal/inference/openai.go:531-586)对 thinking 只做一件事:disableThinking &&
openAIModelSupportsDisableThinking(model) 时注入 enable_thinking=false,而
openAIModelSupportsDisableThinking(:645)只匹配 qwen3/qwq 前缀。
• GLM/Kimi/DeepSeek-V4/Mimo 用的 thinking:{type:disabled} 注入在 applyThinkingDisableControls(:604-618),它只在
applyExplicitThinkingDisableControls 里被调用(:588),而后者只在带 opts 的 chat 请求路径执行(:296-298,需
opts.DisableThinking)。
• 评估调用走的是 ChatCompletionProxier.ChatCompletion(provider_pool_evaluation.go:1125 → openai.go:117-122),只过
sanitize,永远不会执行 applyExplicitThinkingDisableControls。

所以 PR 说的"引擎层已按模型处理"在评估这条路径上不成立。

问题 2 — 对 thinking 系 judge 模型有回归风险。 judge 全部走
source="cloud"(provider_pool_evaluation.go:128/247/568/847),cloud 引擎是
NewOpenAIEngine(handlers_cloud.go:300,disableThinking=true)。删掉硬编码后:

• Qwen3/QwQ judge:安全,sanitize 仍会注入 enable_thinking=false。
• GLM-5.x / DeepSeek-V4 / Kimi / Mimo judge:没有任何 thinking 抑制兜底。这些模型默认开 thinking,思考内容若混进
content,judge 解析直接失败——parseProviderPoolJudge(:1396)和 parseProviderPoolListwiseJudge(:1209)都是
DisallowUnknownFields + 必须精确 EOF 的严格 JSON。失败后进入重试循环(:544-620),每次重试前先 reserve 预算
(:553-561),结果就是评估成功率下降、预算膨胀。当初写死 disable 大概率就是为跨模型统一保证 judge 纯 JSON 输出。

问题 3 — 测试保护被删。 PR 把 provider_pool_evaluation_test.go:294-296 的 thinking 断言删了,等于撤掉了"judge 请求必须
抑制 thinking"这条保护。我刚跑过现状测试(TestProviderPoolEvaluationJudgeRetriesStrictJSONAndAccountsBudget)是过的,
说明现有行为依赖这个断言。

Instead of dropping the thinking field for every judge, keep suppressing
reasoning for model families that accept it (GLM, Kimi, Moonshot,
DeepSeek-V4, Mimo) and omit it for strict OpenAI-compatible endpoints
such as GPT-4.1 mini that reject unknown arguments with 400.
Qwen3/QwQ judges are covered by the engine's enable_thinking=false
injection in sanitizeOpenAIRequestBody, so the field stays off for them.

Expose inference.OpenAIModelUsesThinkingTypeDisabled so the evaluation
judge request can reuse the same per-model-family policy. Add a table
test covering both groups.
@apapi

apapi commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

用最新代码测试了不同的judge model(glm5.2和gpt-4.1 mini),均未发现问题。

@apapi
apapi merged commit 2f071aa into OpenCSGs:main Sep 2, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

配置评估出现问题

2 participants