fix(agent-challenge): supply LLM_MODEL to the evaluated agent - #63
Merged
Conversation
Every Terminal-Bench evaluation scored 0 and every result.json carried "model_name": null. The packaged agent fails closed without a concrete model id: agent run failed: A concrete model id is required: set LLM_MODEL (the measured review harness supplies it under .rules). Three independent layers dropped the model id, so no submission could ever score: 1. the runner never set LLM_MODEL -- the platform only holds it as CHALLENGE_LLM_MODEL (settings.llm_model), which the agent never reads; 2. sanitize_miner_env_for_job stripped a miner-supplied LLM_MODEL because the name is not token/key shaped; 3. AGENT_ENV_ALLOWLIST admitted only LLM_COST_LIMIT and OPENROUTER_API_KEY. The trial died in ~5s per task, before any LLM call, which is why the whole leaderboard sat at zero while OPENROUTER_API_KEY was reaching the agent correctly. _terminal_bench_env now publishes settings.llm_model as LLM_MODEL, ahead of the sanitized miner merge so a miner may override it; the model name joins both the miner product allowlist and the agent sandbox allowlist. A miner can therefore bring their own key and their own model. When an operator blanks llm_model nothing is injected, so the agent still fails loudly rather than running an unmeasured default. Updated pins keep their intent: gateway/URL/host/proxy names stay excluded, secret scrubbing still asserted, and the ex-gateway test now asserts LLM_MODEL comes from settings and that no gateway base_url or token leaks into the job env. Verified: 7 new tests RED -> GREEN; full agent-challenge suite shows zero regressions (26 pre-existing failures on main, 26 with this change, identical sets; 2286 -> 2293 passing).
📝 WalkthroughWalkthroughChangesLLM model environment propagation
Estimated code review effort: 2 (Simple) | ~10 minutes Sequence Diagram(s)sequenceDiagram
participant PlatformSettings
participant terminal_bench_env
participant sanitize_miner_env_for_job
participant filter_agent_env
participant AgentContainer
PlatformSettings->>terminal_bench_env: provide settings.llm_model
terminal_bench_env->>terminal_bench_env: set default LLM_MODEL
terminal_bench_env->>sanitize_miner_env_for_job: validate miner LLM_MODEL
sanitize_miner_env_for_job-->>terminal_bench_env: return accepted override
terminal_bench_env->>filter_agent_env: forward LLM_MODEL
filter_agent_env->>AgentContainer: retain LLM_MODEL
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
packages/challenges/agent-challenge/tests/test_llm_model_reaches_agent.py (1)
48-52: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd regression coverage for a blank platform model.
The PR contract says blank
settings.llm_modelmust leaveLLM_MODELunset, but these tests cover only the nonblank default and miner override. Add a test that sets the setting to""and asserts the key is absent from_terminal_bench_env().🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/challenges/agent-challenge/tests/test_llm_model_reaches_agent.py` around lines 48 - 52, Add a regression test alongside test_runner_injects_platform_llm_model that temporarily sets settings.llm_model to an empty string, calls _terminal_bench_env(), and asserts LLM_MODEL is absent from the returned environment mapping. Restore the setting after the test using the existing test framework’s fixture or cleanup mechanism.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/challenges/agent-challenge/src/agent_challenge/submissions/miner_env.py`:
- Around line 49-52: Remove LLM_MODEL from MINER_ENV_PRODUCT_ALLOWLIST and keep
it out of the secret validation path used by validate_miner_env() and
sanitize_miner_env_for_job(). Handle miner-provided model identifiers through a
separate narrowly scoped model-ID validator or allowlist, preserving rejection
of URL-shaped or endpoint-like values.
---
Nitpick comments:
In `@packages/challenges/agent-challenge/tests/test_llm_model_reaches_agent.py`:
- Around line 48-52: Add a regression test alongside
test_runner_injects_platform_llm_model that temporarily sets settings.llm_model
to an empty string, calls _terminal_bench_env(), and asserts LLM_MODEL is absent
from the returned environment mapping. Restore the setting after the test using
the existing test framework’s fixture or cleanup mechanism.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: a7c14913-ab5b-462b-9f39-0b3610eed45d
📒 Files selected for processing (8)
packages/challenges/agent-challenge/src/agent_challenge/evaluation/own_runner/isolation.pypackages/challenges/agent-challenge/src/agent_challenge/evaluation/runner.pypackages/challenges/agent-challenge/src/agent_challenge/submissions/miner_env.pypackages/challenges/agent-challenge/tests/test_deepseek_via_gateway.pypackages/challenges/agent-challenge/tests/test_llm_model_reaches_agent.pypackages/challenges/agent-challenge/tests/test_miner_env_lock.pypackages/challenges/agent-challenge/tests/test_own_runner_isolation.pypackages/challenges/agent-challenge/tests/test_worker_security.py
Comment on lines
+49
to
+52
| # VAL-LLM-MODEL: a miner may bring their own key *and* their own model | ||
| # id. Not a URL/host/proxy name, and URL-shaped values stay rejected by | ||
| # ``looks_like_url_value``, so this opens no endpoint-injection hole. | ||
| "LLM_MODEL", |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## Locate and inspect miner_env.py"
fd -a 'miner_env\.py$' . | sed 's#^\./##'
file="$(fd 'miner_env\.py$' . | head -n1)"
if [ -n "${file:-}" ]; then
echo "--- $file ($(wc -l < "$file") lines) ---"
cat -n "$file"
fi
echo
echo "## Search for MINER_ENV_PRODUCT_ALLOWLIST and related validators"
rg -n --hidden --glob '*.py' 'MINER_ENV_PRODUCT_ALLOWLIST|is_token_or_key_env_name|looks_like_url_value|LLM_MODEL|model' packages/challenges/agent-challenge/src packages/challenges/agent-challenge || trueRepository: BaseIntelligence/base
Length of output: 50378
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="packages/challenges/agent-challenge/src/agent_challenge/submissions/miner_env.py"
echo "--- miner_env.py ---"
cat -n "$file"
echo
echo "--- relevant call sites ---"
sed -n '150,220p' packages/challenges/agent-challenge/src/agent_challenge/evaluation/own_runner/isolation.py | cat -n
sed -n '1628,1648p' packages/challenges/agent-challenge/src/agent_challenge/evaluation/runner.py | cat -n
sed -n '1988,2002p' packages/challenges/agent-challenge/src/agent_challenge/evaluation/runner.py | cat -n
echo
echo "--- behavioral probe of miner_env validation ---"
python3 - <<'PY'
import sys
sys.path.insert(0, 'packages/challenges/agent-challenge/src')
from agent_challenge.submissions.miner_env import (
MINER_ENV_PRODUCT_ALLOWLIST,
is_token_or_key_env_name,
looks_like_url_value,
)
cases = {
"LLM_MODEL": "x-ai/grok-4.5",
"OPENROUTER_API_KEY": "sk-or-prod-secret",
"LLM_MODEL_URL_HOST": "https://evil.internal.example/model",
"LLM_MODEL_BEARER": "Bearer token=123",
"X_VALID_KEY": "abc123",
"PORT": "80",
}
for name, value in case := cases.items():
print(name, {
"is_allowed_token_or_key": is_token_or_key_env_name(name),
"looks_like_url": looks_like_url_value(value),
"upper_allowed": name.upper() in MINER_ENV_PRODUCT_ALLOWLIST,
"value": value,
})
PYRepository: BaseIntelligence/base
Length of output: 13298
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "--- behavioral probe of miner_env validation ---"
python3 - <<'PY'
import sys
sys.path.insert(0, 'packages/challenges/agent-challenge/src')
from agent_challenge.submissions.miner_env import (
MINER_ENV_PRODUCT_ALLOWLIST,
is_token_or_key_env_name,
looks_like_url_value,
)
cases = {
"LLM_MODEL": "x-ai/grok-4.5",
"LLM_MODEL_SECRET": "x-ai/grok-4.5",
"OPENROUTER_API_KEY": "sk-or-prod-secret",
"LLM_MODEL_URL_HOST": "https://evil.internal.example/model",
"LLM_MODEL_BEARER": "Bearer token=123",
"X_VALID_KEY": "abc123",
"PORT": "80",
}
for name, value in cases.items():
print(name, {
"is_allowed_token_or_key": is_token_or_key_env_name(name),
"looks_like_url": looks_like_url_value(value),
"upper_allowed": name.upper() in MINER_ENV_PRODUCT_ALLOWLIST,
"value": value,
})
PY
echo
echo "--- submit/env route / API route env handling slices ---"
file="packages/challenges/agent-challenge/src/agent_challenge/api/routes.py"
sed -n '2820,2890p' "$file" | cat -n
sed -n '2898,2918p' "$file" | cat -n
rg -n "validate_miner_env|sanitize_miner_env_for_job|miner_env|SUBMIT.*ENV|env" "$file" -A 4 -B 4Repository: BaseIntelligence/base
Length of output: 33243
Do not add LLM_MODEL to the secret allowlist.
MINER_ENV_PRODUCT_ALLOWLIST is the “API key / token / measured product secret” gate; LLM_MODEL is an OpenRouter model identifier, not a secret. Keeping it here lets miners submit and persist arbitrary model names through validate_miner_env()/sanitize_miner_env_for_job(); use a separate narrowly scoped model-id validator/allowlist instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@packages/challenges/agent-challenge/src/agent_challenge/submissions/miner_env.py`
around lines 49 - 52, Remove LLM_MODEL from MINER_ENV_PRODUCT_ALLOWLIST and keep
it out of the secret validation path used by validate_miner_env() and
sanitize_miner_env_for_job(). Handle miner-provided model identifiers through a
separate narrowly scoped model-ID validator or allowlist, preserving rejection
of URL-shaped or endpoint-like values.
Source: Coding guidelines
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every Terminal-Bench evaluation scored 0/30 and every
result.jsoncarried"model_name": null. The packaged agent fails closed without a concrete model id:This error never reached the database — it is written to
exception.txtinside the trial dir, while the job log events are truncated at 16 KB. Diagnosed by runningown_runner_backendby hand on the prod validator with the app's own environment.Root cause — three independent layers
_terminal_bench_envLLM_MODEL; the platform only holdsCHALLENGE_LLM_MODEL(settings.llm_model), which the agent never readssanitize_miner_env_for_jobLLM_MODEL— the name is not token/key shapedAGENT_ENV_ALLOWLISTLLM_COST_LIMITandOPENROUTER_API_KEYEmpirically confirmed:
OPENROUTER_API_KEYwas reaching the agent correctly the whole time — only the model id was missing, so each trial died in ~5 s before any LLM call. It was structurally impossible for any submission to score.Fix
_terminal_bench_envpublishessettings.llm_modelasLLM_MODEL, before the sanitized miner merge so a miner can override it.LLM_MODELjoinsMINER_ENV_PRODUCT_ALLOWLIST— a miner may bring their own key and their own model.LLM_MODELjoinsAGENT_ENV_ALLOWLISTso it survivesfilter_agent_env.llm_model, nothing is injected: the agent still fails loudly rather than running an unmeasured default.Security invariants preserved
assert_no_untrusted_secretguards untouched;LLM_MODELis not secret-shaped.LLM_MODELcomes from settings and that no gatewaybase_url/ token leaks — stronger than the previous blanket ban, which only existed because the gateway used to be its sole source.Verification
agent-challengesuite: zero regressions — 26 pre-existing failures onmain, 26 with this change, identical sets; passing 2286 → 2293.LLM_MODELat runtime moved the trial from dying in 5 s to running task containers for minutes (alexgshaw-cancel-async-tasks-...-tmuxup 2 and 8 minutes).Summary by CodeRabbit
LLM_MODEL.