Skip to content

fix(web): ship placeholder judge model so a clean clone resolves & runs - #4

Open
kongyoongkeong-lab wants to merge 1 commit into
Tencent:mainfrom
kongyoongkeong-lab:fix/web-judge-model-missing
Open

fix(web): ship placeholder judge model so a clean clone resolves & runs#4
kongyoongkeong-lab wants to merge 1 commit into
Tencent:mainfrom
kongyoongkeong-lab:fix/web-judge-model-missing

Conversation

@kongyoongkeong-lab

@kongyoongkeong-lab kongyoongkeong-lab commented Jul 27, 2026

Copy link
Copy Markdown

Summary

The Web benchmark (configs/bench/wb-bench-web-v1.0.yaml) pins llm_judge.model: kimi-k2.7-think by default, but configs/models/** is gitignored (only _template.model.yaml ships). On a clean clone, resolve_manifest therefore fails every Web run with:

llm_judge.model='kimi-k2.7-think' does not resolve to a model config
at .../configs/models/kimi-k2.7-think.yaml

All 70 Web tasks contain llm/vlm judge items, and the in-container CompositeVerifier raises when the verifier LLM route is not configured (route_ready=False), so the Web track is currently unrunnable out of the box.

Fix

  • Un-ignore and ship configs/models/kimi-k2.7-think.yaml as an env-only placeholder (mirrors _template.model.yaml). Credentials still live in .env (KIMI_BASE_URL / KIMI_API_KEY); the file names only the env vars to read.
  • The web bench default (enabled: true, model: kimi-k2.7-think) is left intact — this only restores the model file the default points at, so the LLM/VLM judge route resolves and the verifier stops raising.

Users with a Moonshot/Kimi backend fill the two .env vars and run as before. The slug now resolves cleanly; a missing KIMI_BASE_URL surfaces a clear runtime error instead of a silent manifest-resolution failure.

Test plan

  • resolve_manifest no longer reports the missing-model error for a Web job (model slug now resolves to a config).
  • Placeholder kimi-k2.7-think.yaml tracks (gitignore exception added).
  • Maintainer: confirm the placeholder shape matches the intended Kimi/OpenAI-compatible endpoint (no key/credential shipped).

Notes / alternatives considered

  • Simply changing the web default to <model-slug> + enabled: false passes the resolver but breaks scoring: the verifier raises on the 70 tasks that require an LLM/VLM route. Keeping enabled: true and restoring the model file is the minimal change that keeps the Web track actually runnable.
  • If the project would rather not ship a Kimi-specific file, an alternative is to document "Web requires a local kimi-k2.7-think.yaml (cp from _template)" in the README and relax the gitignore similarly — happy to adjust.

The Web benchmark (configs/bench/wb-bench-web-v1.0.yaml) pins
llm_judge.model: kimi-k2.7-think by default, but configs/models/**
is gitignored (only _template.model.yaml is shipped). On a clean
clone, resolve_manifest therefore fails every Web run with:

  llm_judge.model='kimi-k2.7-think' does not resolve to a model config
  at .../configs/models/kimi-k2.7-think.yaml

All 70 Web tasks contain llm/vlm judge items, and the in-container
CompositeVerifier raises when the verifier LLM route is not configured,
so the Web track is currently unrunnable out of the box.

Fix: un-ignore and ship configs/models/kimi-k2.7-think.yaml as an
env-only placeholder (mirrors _template.model.yaml). Credentials still
come from .env (KIMI_BASE_URL / KIMI_API_KEY). Real backends fill those
two vars; the slug now resolves so the LLM/VLM judge route is wired up.

The web bench default (enabled: true, model: kimi-k2.7-think) is left
intact - this only restores the model file the default points at.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant