3/3 Add harness-engineering-bench (GAIA + benchmarks)#46
Open
varunursekar wants to merge 30 commits into
Open
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+23
to
+24
| docker pull yimjk/ale-bench:cpp20-202301 | ||
| docker tag yimjk/ale-bench:cpp20-202301 ale-bench:cpp20-202301 |
There was a problem hiding this comment.
cpp20-202301) without a digest pin. Docker Hub tags can be silently overwritten, so two scoring runs could pull different binaries under the same tag — directly threatening benchmark reproducibility. In the worst case, a compromised yimjk account could push a malicious image that gets pulled at evaluation time. Pinning to the image digest locks the exact bytes regardless of tag mutations.
Suggested change
| docker pull yimjk/ale-bench:cpp20-202301 | |
| docker tag yimjk/ale-bench:cpp20-202301 ale-bench:cpp20-202301 | |
| docker pull yimjk/ale-bench:cpp20-202301@sha256:<DIGEST> | |
| docker tag yimjk/ale-bench:cpp20-202301@sha256:<DIGEST> ale-bench:cpp20-202301 |
Prompt To Fix With AI
This is a comment left during a code review.
Path: harness-engineering-bench/ale-bench/environment/sidecar/dind-entrypoint.sh
Line: 23-24
Comment:
The judge image is pulled by mutable tag (`cpp20-202301`) without a digest pin. Docker Hub tags can be silently overwritten, so two scoring runs could pull different binaries under the same tag — directly threatening benchmark reproducibility. In the worst case, a compromised `yimjk` account could push a malicious image that gets pulled at evaluation time. Pinning to the image digest locks the exact bytes regardless of tag mutations.
```suggestion
docker pull yimjk/ale-bench:cpp20-202301@sha256:<DIGEST>
docker tag yimjk/ale-bench:cpp20-202301@sha256:<DIGEST> ale-bench:cpp20-202301
```
How can I resolve this? If you propose a fix, please make it concise.
varunursekar
force-pushed
the
pr3-harness-bench
branch
3 times, most recently
from
July 23, 2026 20:00
e0a1efb to
ab4c52f
Compare
varunursekar
force-pushed
the
pr3-harness-bench
branch
2 times, most recently
from
July 23, 2026 22:58
573c1ac to
c86351e
Compare
| close() | ||
|
|
||
| judge = str(getattr(result, "overall_judge_result", "")) | ||
| accepted = 1.0 if judge.upper().endswith("ACCEPTED") else 0.0 |
There was a problem hiding this comment.
endswith("ACCEPTED") will incorrectly match any string that ends with "ACCEPTED" — most notably "NOT_ACCEPTED" (or an enum-style repr like "JudgeResult.NOT_ACCEPTED"). If ALE-Bench returns such a value for a failing submission, the accepted metric would be written as 1.0 for a rejected run. Using exact equality is unambiguous regardless of what the library returns.
Suggested change
| accepted = 1.0 if judge.upper().endswith("ACCEPTED") else 0.0 | |
| accepted = 1.0 if judge.upper() == "ACCEPTED" else 0.0 |
Prompt To Fix With AI
This is a comment left during a code review.
Path: harness-engineering-bench/ale-bench/environment/sidecar/harness/score.py
Line: 83
Comment:
`endswith("ACCEPTED")` will incorrectly match any string that ends with "ACCEPTED" — most notably "NOT_ACCEPTED" (or an enum-style repr like `"JudgeResult.NOT_ACCEPTED"`). If ALE-Bench returns such a value for a failing submission, the `accepted` metric would be written as `1.0` for a rejected run. Using exact equality is unambiguous regardless of what the library returns.
```suggestion
accepted = 1.0 if judge.upper() == "ACCEPTED" else 0.0
```
How can I resolve this? If you propose a fix, please make it concise.
varunursekar
force-pushed
the
pr3-harness-bench
branch
3 times, most recently
from
July 24, 2026 03:49
745ed8e to
6788d76
Compare
varunursekar
force-pushed
the
pr3-harness-bench
branch
4 times, most recently
from
July 24, 2026 06:41
214f1aa to
a2fcd19
Compare
shehabyasser-scale
approved these changes
Jul 24, 2026
Collaborator
|
Approve with nits. The split is the part I care most about for experiment validity, and it checks out. GAIA split is genuinely leak-safe (verified):
One real defect (baseline quality, not integrity): the tau3 baseline drops conversation history after every plain-text turn. One thing to note for readers: every build.yaml ships |
varunursekar
force-pushed
the
pr3-harness-bench
branch
4 times, most recently
from
July 24, 2026 22:28
768de08 to
ad2c081
Compare
Comment on lines
+310
to
+322
| for turn in range(1, MAX_TURNS + 1): | ||
| turns = turn | ||
| kwargs: dict[str, Any] = { | ||
| "model": self._api_model, | ||
| "messages": messages, | ||
| "tools": openai_tools, | ||
| "max_tokens": 8_000, | ||
| } | ||
| # OpenAI reasoning models accept reasoning_effort; other providers | ||
| # (e.g. Fireworks-served open models) reject it. | ||
| if "fireworks" not in self._api_model: | ||
| kwargs["reasoning_effort"] = "medium" | ||
| response = await client.chat.completions.create(**kwargs) |
There was a problem hiding this comment.
parallel_tool_calls not disabled while only calls[0] is processed
The tau3 agent explicitly only acts on the first tool call (call = calls[0], line 347), and only that call's id appears in the recorded assistant message. The loop kwargs, however, do not include "parallel_tool_calls": False, unlike the GAIA and OfficeQA agents which both set it explicitly. When running against an OpenAI reasoning model with reasoning_effort, the model may return multiple tool calls in one turn. calls[1:] are then silently dropped — neither executed nor echoed back to the model — so the model proceeds believing it made N calls while only the first was actually dispatched. For a customer-service agent that might simultaneously call verify_identity and send_message_to_user, this silently skips the second action and produces incorrect conversation state.
Suggested change
| for turn in range(1, MAX_TURNS + 1): | |
| turns = turn | |
| kwargs: dict[str, Any] = { | |
| "model": self._api_model, | |
| "messages": messages, | |
| "tools": openai_tools, | |
| "max_tokens": 8_000, | |
| } | |
| # OpenAI reasoning models accept reasoning_effort; other providers | |
| # (e.g. Fireworks-served open models) reject it. | |
| if "fireworks" not in self._api_model: | |
| kwargs["reasoning_effort"] = "medium" | |
| response = await client.chat.completions.create(**kwargs) | |
| for turn in range(1, MAX_TURNS + 1): | |
| turns = turn | |
| kwargs: dict[str, Any] = { | |
| "model": self._api_model, | |
| "messages": messages, | |
| "tools": openai_tools, | |
| "max_tokens": 8_000, | |
| "parallel_tool_calls": False, | |
| } |
Prompt To Fix With AI
This is a comment left during a code review.
Path: harness-engineering-bench/tau3/baseline/target/src/tau3_agent/agent.py
Line: 310-322
Comment:
**`parallel_tool_calls` not disabled while only `calls[0]` is processed**
The tau3 agent explicitly only acts on the first tool call (`call = calls[0]`, line 347), and only that call's id appears in the recorded assistant message. The loop kwargs, however, do not include `"parallel_tool_calls": False`, unlike the GAIA and OfficeQA agents which both set it explicitly. When running against an OpenAI reasoning model with `reasoning_effort`, the model may return multiple tool calls in one turn. `calls[1:]` are then silently dropped — neither executed nor echoed back to the model — so the model proceeds believing it made N calls while only the first was actually dispatched. For a customer-service agent that might simultaneously call `verify_identity` and `send_message_to_user`, this silently skips the second action and produces incorrect conversation state.
```suggestion
for turn in range(1, MAX_TURNS + 1):
turns = turn
kwargs: dict[str, Any] = {
"model": self._api_model,
"messages": messages,
"tools": openai_tools,
"max_tokens": 8_000,
"parallel_tool_calls": False,
}
```
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+124
to
+131
| if record.get("scope") not in ("evaluation", "finalization"): | ||
| continue | ||
| evaluation_id = record.get("attribution") | ||
| snippet = record.get("root_snippet") or recover_snippet(record) | ||
| thread_key = record.get("thread_id") or ( | ||
| "snippet:" + normalize(snippet or "")[:160] or "unattributed" | ||
| ) | ||
| thread = per_evaluation[evaluation_id].setdefault( |
There was a problem hiding this comment.
None evaluation key crashes sorted() on records missing attribution
evaluation_id = record.get("attribution") returns None when a record with scope == "evaluation" or scope == "finalization" has no attribution field. This None is then used as a defaultdict key, which is silently valid. However, sorted(threads_by_evaluation.items()) in main() will raise TypeError: '<' not supported between instances of 'NoneType' and 'str' the moment any such record is present — crashing the entire accounting run. The docstring explicitly mentions "salvage copies" as supported inputs, which may not have full attribution annotations.
Prompt To Fix With AI
This is a comment left during a code review.
Path: harness-engineering-bench/scripts/per_trial_tokens.py
Line: 124-131
Comment:
**`None` evaluation key crashes `sorted()` on records missing `attribution`**
`evaluation_id = record.get("attribution")` returns `None` when a record with `scope == "evaluation"` or `scope == "finalization"` has no `attribution` field. This `None` is then used as a `defaultdict` key, which is silently valid. However, `sorted(threads_by_evaluation.items())` in `main()` will raise `TypeError: '<' not supported between instances of 'NoneType' and 'str'` the moment any such record is present — crashing the entire accounting run. The docstring explicitly mentions "salvage copies" as supported inputs, which may not have full attribution annotations.
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+255
to
+278
| for call in calls: | ||
| try: | ||
| arguments = json.loads(call.function.arguments) | ||
| except json.JSONDecodeError as error: | ||
| result: dict[str, Any] = {"error": f"invalid arguments: {error}"} | ||
| image_url = None | ||
| else: | ||
| image_url = None | ||
| if call.function.name == "run_shell": | ||
| result = await self._run_shell( | ||
| environment, arguments["command"] | ||
| ) | ||
| elif call.function.name == "read_image": | ||
| result, image_url = await self._read_image( | ||
| environment, arguments["path"] | ||
| ) | ||
| elif call.function.name == "submit_answer": | ||
| await self._submit(environment, arguments["answer"]) | ||
| result = {"submitted": True} | ||
| submitted = True | ||
| else: | ||
| result = {"error": f"unknown tool: {call.function.name}"} | ||
| self._trace( | ||
| {"turn": turn, "tool": call.function.name, "result": result} |
There was a problem hiding this comment.
KeyError not caught when model omits required argument
The try block only catches json.JSONDecodeError, but accesses like arguments["command"], arguments["path"], and arguments["answer"] can raise KeyError if the model returns syntactically valid JSON that omits a required field. Without "strict": True in the Chat Completions tool schema (unlike the GAIA agent which uses it), the API does not guarantee required fields are present. A missing argument crashes the entire trial with an unhandled exception rather than feeding an error back to the model. The BrowseComp-Plus agent in this same PR guards against this correctly with except (json.JSONDecodeError, KeyError, TypeError, ValueError). The same gap exists in swe-atlas-qna/baseline/target/src/atlas_agent/agent.py lines 234–260.
Prompt To Fix With AI
This is a comment left during a code review.
Path: harness-engineering-bench/officeqa/baseline/target/src/officeqa_agent/agent.py
Line: 255-278
Comment:
**`KeyError` not caught when model omits required argument**
The `try` block only catches `json.JSONDecodeError`, but accesses like `arguments["command"]`, `arguments["path"]`, and `arguments["answer"]` can raise `KeyError` if the model returns syntactically valid JSON that omits a required field. Without `"strict": True` in the Chat Completions tool schema (unlike the GAIA agent which uses it), the API does not guarantee required fields are present. A missing argument crashes the entire trial with an unhandled exception rather than feeding an error back to the model. The BrowseComp-Plus agent in this same PR guards against this correctly with `except (json.JSONDecodeError, KeyError, TypeError, ValueError)`. The same gap exists in `swe-atlas-qna/baseline/target/src/atlas_agent/agent.py` lines 234–260.
How can I resolve this? If you propose a fix, please make it concise.Bring the benchmark harness with each benchmark as a top-level sibling of gaia/ (ale-bench, officeqa, swe-atlas-qna, tau3), flattening the former candidates/ grouping.
The floor gates a ship on a validation comparison while gaia's reward is on the test partition (different distributions), so keep it opt-in / off. Now that the harbor default is false this is explicit-for-clarity.
Now that the sidecar image pre-warms the harness uv cache, the inner eval runs as the unprivileged harness user again, so the candidate harness cannot read the held-out records, budgets, or admin token.
Move each benchmark to harbor 0.20.0 in lockstep so the eval resolves: build.yaml harbor_requirement (the evaluator's --with harbor[modal]) and target/pyproject.toml (the candidate's own harbor dep) must match, else `uv run --project <target> --with harbor[modal]==0.20.0` is unsatisfiable. uv.lock regenerated with `uv lock` (not hand-edited). Covers gaia, officeqa, swe-atlas-qna, tau3. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the pr2 rename: instructions, READMEs, and the ale-bench reference solution now teach `evals run/status/result/submit` and the .evals/tasks + .evals/results paths; baseline gitignores cover .evals/. Compiled trees are build outputs (not committed) and pick this up on rebuild. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All four vero-compiled benchmarks pass --ek app_name=harness-engineering-bench so their Modal sandboxes are grouped under one app instead of __harbor__. Docker inner environments ignore the kwarg.
The task's verifier runs inside the Modal task container and templates EVAL_BASE_URL from OPENAI_API_BASE, unreachable-from-Modal gateway URLs aside. Enable task_services_use_upstream so the judge gets the real upstream on OPENAI_*, and point the baseline agent at the metered gateway via VERO_AGENT_INFERENCE_* (falling back to OPENAI_* locally).
The Modal app-name commit added a second extra_harbor_args key; YAML last-key-wins silently dropped --no-force-build, which would have force-built the guard Dockerfile instead of pulling the prebuilt corpus image.
Killing the outer harbor process leaves the compose topology and any in-flight Modal sandboxes running (each layer's cleanup lives in the layer above). Cap sandbox idle lifetime at 1h via --ek, and add scripts/cleanup_orphans.sh to tear down task__* containers and terminate sandboxes in the dedicated Modal app.
Across gaia/officeqa/swe-atlas-qna/tau3: 100-run budgets and 4x-pass
case budgets on both agent partitions, submit-mode selection,
baseline_floor off, 16k feedback, parameterized inner environment
(${inner_env:-modal}), and verifier_timeout = 2x timeout_seconds.
swe-atlas-qna and tau3 get explicit case/task-agent timeouts derived
from their datasets' declared agent timeouts (enforced budgets 1800s
and 900s, provisional until empirical wall_seconds data). harness_user
is explicit on gaia/officeqa; tau3 and swe-atlas-qna stay unisolated
pending off-env delivery of task-service credentials. Also fix the
vendor script's stale pre-restructure path in its usage comment.
Root cause of every swe-atlas trial dying at setup: the task images set ENTRYPOINT ["/bin/bash"], Modal prepends it to the sandbox command, so harbor's default keepalive ["sh","-c","sleep infinity"] ran as 'bash sh -c ...' - bash interprets the sh binary as a script and exits 126, killing the sandbox before the first exec. Pass bash's own args via the JSON-decoded keepalive env kwarg instead. Validated end-to-end on Modal (3 dev trials, 379-602s wall). CONFIGURATION.md records the empirical probe timings behind the provisional case timeouts.
scripts/per_trial_tokens.py reads a session's gateway request log and attributes token/latency to Harbor trials: by the gateway's stamped thread_id when present (request_log.attribution builds → full coverage incl. stateful chains), else a legacy fallback that recovers each conversation's root turn (labelable via --tasks-dir) and residualizes unrecoverable follow-ups. Reports gateway-total vs attributed vs agent-reported per evaluation. Stdlib-only; tests cover both paths.
…r at turn cap Chat Completions is the only universally-supported inference surface — the OpenAI Responses API doesn't translate cleanly to non-OpenAI models via litellm (parallel_tool_calls rejected, output_text crashes on a null message part). Rewrite AtlasAgent's loop to stateless chat messages + tool_calls, validated cross-provider (gpt-5.4-mini and fireworks_ai/gpt-oss-120b both run the multi-turn tool loop). Also: at the turn cap, force one final tool-free answer instead of raising, so a budget-exhausted trial scores best-effort rather than losing the case.
Same port as swe-atlas: stateless chat messages + tool_calls, works across providers (validated cross-provider on gpt-5.4-mini and fireworks_ai/gpt-oss-120b). officeqa drops the hosted web_search tool (closed-corpus benchmark, and chat/completions has no hosted-tool equivalent) and moves read_image feedback to chat image_url content. tau3 keeps its host-side MCP loop; the stateless history also fixes the previous_response_id reset that dropped context after plain-text turns, and the MCP session id is now charset-validated before shell interpolation (greptile injection finding).
Mirrors the swe-atlas fix: at MAX_TURNS force one tool-free completion for the best answer and submit it, so a budget-exhausted case scores best-effort rather than being lost with no answer recorded.
A fifth harness-engineering benchmark: optimize a retrieval + reasoning agent over BrowseComp-Plus's fixed corpus and canonical BM25 index, graded by a semantic answer judge (gpt-4.1). Task packages are generated from the encrypted upstream dataset by scripts/build_tasks.py and are gitignored (they contain decrypted questions/answers); the upstream repo is pinned as a submodule and the 2.2 GB index is pulled at image-build time. Wired consistently with the normalized suite: 33/66/66 stratified split, 4-pass budgets, submit-mode selection, harness_user null + task_services_use_upstream for the in-container judge. Baseline agent uses Chat Completions (works across providers) and forces a best-effort answer at the turn cap. The task image installs a full JDK with JAVA_HOME resolved from it, so pyserini's Lucene BM25 searcher works (a headless JRE with JAVA_HOME unset left jnius unable to locate the JVM).
… succeeds pyserini 1.2.0 constructs an openai.OpenAI() client at import time for its unused OpenAI encoders; BM25/Lucene search never calls OpenAI, so set a placeholder key in search.py before the import.
The task.toml had no verifier.env, so the gpt-4.1 answer judge ran without OPENAI_API_KEY and scored every case 0 (Missing credentials). Template the upstream creds and judge model into the verifier phase.
…las) From a 10-trial per-benchmark probe of gpt-5.4-mini vs gpt-oss-120b vs deepseek-v4-flash: deepseek matches/beats both on tau3 (0.875) and is ~2-3x gpt-oss on the grounded-reasoning benchmarks (officeqa/browsecomp 0.60) at roughly gpt-oss cost and far below mini. It is weaker only on swe-atlas (0.30 vs gpt-oss 0.59 mean rubric), which is pinned to gpt-oss-120b — cheaper and stronger there. Updates each benchmark's target `model` and the evaluation-scope `allowed_models` (kept with the fireworks_ai/ prefix, since agents strip only the openai/ prefix and the gateway matches the sent model exactly). Producer scopes and tau3's user/assertion models are unchanged. Also hardens the swe-atlas atlas agent: a shell command emitting non-UTF-8 (binary) output raised UnicodeDecodeError and killed the whole trial; it now returns a tool error so the model can retry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… relock - .gitignore: remove the paper-planning benchmark-scoping.md ignore - README.md: correct candidate paths (candidates/ -> top level), add browsecomp-plus, drop the stale nested table - vero/uv.lock: relock Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fireworks' shared-org rate limit surfaces as a RateLimitError inside the agent's OpenAI client; harbor only retries infra/env failures, not LLM 429s, so a burst lost trials. The SDK's default 2 retries were exhausted under sustained load. Bump to 8 so transient 429s back off in-place instead of failing the trial (durable fix is gateway-level retry, tracked separately). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
varunursekar
force-pushed
the
pr3-harness-bench
branch
from
July 25, 2026 19:52
741bfa6 to
95fed89
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of three stacked PRs splitting #41. Base:
pr2-add-vero.Adds
harness-engineering-bench/with each benchmark as a top-level sibling (gaia,ale-bench,officeqa,swe-atlas-qna,tau3) + scripts. Pure adds (71 files).Union of 1/3 + 2/3 + 3/3 ==
v0.5plus alegacy/archive (verified).🤖 Generated with Claude Code
Greptile Summary
This PR adds the
harness-engineering-bench/directory (71 new files) containing five benchmark integrations — GAIA, ALE-Bench, OfficeQA, SWE-Atlas-QnA, and Tau3 — plus BrowseComp-Plus and shared scripts. Each benchmark ships a baseline agent, partition manifests, and abuild.yamlwiring the evaluation into vero's optimization loop.RuntimeErrorat the budget limit (see existing comment).score.py) driven by a Docker-based judge; BrowseComp-Plus uses an LLM judge inevaluate.pywith three regex fallback patterns and explicitparse_errorlogging.partition_dataset.pyimplements deterministic stratified splits for SWE-Atlas-QnA and Tau3;build_tasks.pydoes the same for BrowseComp-Plus using a pinned upstream submodule commit.Confidence Score: 3/5
Several open defects from earlier review rounds remain unaddressed — notably the Tau3 shell injection via unsanitized MCP session ID, missing context.metadata on Tau3 max-turns exhaustion, per_trial_tokens.py sorted() crash on records without attribution, and the GAIA agent's hard crash instead of a best-effort answer when the turn budget runs out.
The new benchmark infrastructure is well-structured overall and the BrowseComp-Plus, OfficeQA, and SWE-Atlas agents handle turn exhaustion correctly. However, a number of the previously flagged defects — shell injection in the Tau3 agent, the per_trial_tokens.py crash path, and the GAIA RuntimeError on budget exhaustion — are still present in the head SHA and directly affect correctness and data integrity of evaluation runs.
Files Needing Attention: harness-engineering-bench/tau3/baseline/target/src/tau3_agent/agent.py (shell injection + missing metadata), harness-engineering-bench/scripts/per_trial_tokens.py (sorted() crash), harness-engineering-bench/gaia/baseline/target/src/gaia_agent/agent.py (no fallback on turn exhaustion)
Important Files Changed
Sequence Diagram
sequenceDiagram participant V as vero optimizer participant S as EvaluationSidecar participant A as BaselineAgent (e.g. GaiaAgent) participant E as Harbor Environment participant LLM as LLM (via inference gateway) participant J as Verifier / Judge V->>S: push candidate (git transport) S->>A: run(instruction, environment, context) loop up to MAX_TURNS A->>LLM: API call (tools / messages) LLM-->>A: response (tool_calls or text) alt tool call A->>E: exec / upload_file E-->>A: result else text answer A->>E: upload /app/answer.txt A-->>S: context.metadata set, break end end S->>J: verify best candidate (test partition) J-->>S: reward score S-->>V: EvaluationResult (score, metrics)Reviews (33): Last reviewed commit: "nit: style" | Re-trigger Greptile