feat(tests): route checker LLM through litellm gpt-5.6-luna via coder-eval 0.11.5

[akshaylive]

Bump coder-eval to 0.11.5 and set checker_context.api_route defaults (default/smoke/smoke-windows/nightly/activation experiments) so the success-checker/judge calls litellm -> gpt-5.6-luna instead of the built-in default, resolving credentials from CODEX_BASE_URL/CODEX_API_KEY.

[github-actions]

Claude finished @akshaylive's task in 2m 9s —— View job

---

PR Review: Route checker LLM through litellm gpt-5.6-luna

• Read review criteria and project conventions
• Read full diff
• Read each changed file in full
• Check for secrets, cross-skill refs, and link validity
• Post review findings

---

Summary

Bumps coder-eval from 0.11.2 to 0.11.3 and adds an identical checker_context.api_route block to 5 experiment files (default, smoke, smoke-windows, nightly, activation) so the success-checker/judge routes through litellm → gpt-5.6-luna using CODEX_BASE_URL/CODEX_API_KEY env vars.

Change-by-Change Review

1. tests/.coder-eval-version

OK — Clean version bump from 0.11.2 → 0.11.3.

2. tests/experiments/default.yaml

OK — checker_context block inserted at line 26, correctly placed between agent: and post_run:. The env var names (CODEX_BASE_URL, CODEX_API_KEY) reference variable names, not secrets. This is a tempdir-based experiment so env vars are inherited from the host — no env_passthrough_extra needed.

3. tests/experiments/smoke.yaml

OK — Identical checker_context block. CODEX_API_KEY and CODEX_BASE_URL are already listed in env_passthrough_extra (lines 29–30), so the docker sandbox will have access to these credentials. Consistent.

4. tests/experiments/smoke-windows.yaml

OK — Identical checker_context block. Tempdir driver — env vars inherited from host. No issues.

5. tests/experiments/nightly.yaml

OK — Identical checker_context block. CODEX_API_KEY and CODEX_BASE_URL are already in env_passthrough_extra (lines 29–30). Consistent with smoke.yaml.

6. tests/experiments/activation.yaml

OK — Identical checker_context block. No explicit sandbox driver (defaults to tempdir), so env vars are inherited. No issues.

What's Missing

Severity: Low — Three other active experiment files lack checker_context and will continue using the built-in default checker model:

• tests/experiments/flow-v2-preview.yaml — active preview experiment (docker driver)
• tests/experiments/same-ground-headtohead.yaml — campaign harness (docker driver)
• tests/experiments/skill-comparison-template.yaml — template for ad-hoc comparisons

This may be intentional (those experiments may use different infra or aren't run against the same judge criteria), but worth confirming. If flow-v2-preview.yaml runs with docker and references CODEX_* env vars elsewhere, it would need env_passthrough_extra entries for CODEX_BASE_URL/CODEX_API_KEY in addition to the checker_context block (it currently doesn't pass those vars through).

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No skill frontmatter changed
E2E Tests	OK	Experiment config only — no task changes
Skill Body	N/A	No skill content changed
References & Assets	N/A	No references changed
Repo Hygiene	OK	No secrets (env var names only), scoped changes, no cross-skill refs

Issues for Manual Review

• Confirm whether flow-v2-preview.yaml and same-ground-headtohead.yaml should also route through litellm → gpt-5.6-luna, or if they intentionally use different checker infra.
• Verify that the api_version: "2024-05-01" parameter is correct for the litellm/gpt-5.6-luna endpoint being targeted.

Conclusion

Clean, well-scoped change. The checker_context block is identical across all 5 files, env var passthrough is already configured in the docker-based experiments, and no secrets are committed. Approve — only a minor question about whether the remaining experiment files need the same treatment.

--- |

[bai-uipath]

LGTM on the direction, but this has never run e2e against a real task. CI died on a missing import before a single judge call went over the wire.

• Run one real task e2e before merge. default.yaml is tempdir, so the checker runs on the host: install the extra locally and run the data-fabric choiceset task that errored in CI, plus one simulation task to confirm the reject. Nothing else proves the call shape, and the bare gpt-5.6-luna id carries no provider prefix where the upstream docs do.

• The litellm extra is installed nowhere, and this repo can't fix it. Under the docker driver the checker runs in-container, so the agent image needs it, as do the coder_eval_uipath host venvs and the skills workflow installs. Fix: --extra litellm in coder_eval's agent Dockerfile, release, re-pin.

• Every simulation.enabled task hard-errors at route resolution. The orchestrator rejects a litellm eval route alongside simulation or agent_judge, so 318 of 1231 tasks die before any criterion runs, roughly 30% of the suite counted as misses. Fix: scope it at the task layer to the 54 llm_judge tasks without simulation.

Minor: api_version is verbatim from the upstream doc example and CODEX_API_VERSION is set nowhere in the pipelines; the activation experiment has no llm_judge criteria, so its block is dead.

[uipreliga]

Fix what you agree with and 🚢