Native eval accepts fully invalid harness runs

[vincentkoc]

What happened

Native eval runs can be reported as completed and included in aggregate scoring
when every trial failed at the harness, infrastructure, or verifier layer.
OpenClaw exit codes 70/71 are a concrete example: verifier rewards may still be
present, while the benchmark execution itself was invalid.

Fleet recovery also inferred a zero exit code from result-file count plus a
final checkpoint marker, even when the archived run exit status was unknown.

Expected behaviour

Preserve verifier rewards for diagnosis, but mark fully invalid runs as
rejected and exclude them from score-eligible aggregates. Fleet completion
must require an explicit archived run exit status.

Steps to reproduce

# Run a native OpenClaw campaign where every trial exits 71 after the verifier
# has produced reward files, then aggregate the exported run.
python -m scripts.native_eval.aggregate <jobs-root> <summary-dir>

Relevant output

NonZeroAgentExitCodeError: Agent exited with code 71

Before the fix, a full set of reward files could still produce a completed,
eligible run despite all trials ending with the harness-specific failure.

Environment

• Python version: 3.12
• OS: Linux
• ClawBench commit: 884dd1b

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed July 30, 2026, 4:38 AM ET / 08:38 UTC.

Summary
Keep this issue open. It is member-authored and an open linked pull request directly implements the reported rejection and fleet-recovery behavior, so closing before that candidate is reviewed and merged would be premature.

Reproducibility: no. the report provides a specific native-campaign and aggregation path, but a current-main campaign was not run during this read-only review. The linked pull request's stated focused test run is useful supplemental evidence, not a reproduced issue result.

Root-cause cluster
Relationship: canonical
Canonical: #63
Summary: This issue is the canonical report for the native-evaluation acceptance defect; the linked open pull request is its direct implementation candidate.

Members:

• fixed_by_candidate: fix(native-eval): reject invalid harness runs #64 - The pull request explicitly targets this issue's fully-invalid-run rejection and fleet-recovery requirements, but is not merged.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Ways to help us reproduce this

• Add expected vs actual behavior.
• Share version, platform, channel/provider, and relevant config details.

Next step
The linked open pull request already owns the implementation path, and this member-authored issue should remain available for maintainer review rather than receive a parallel repair dispatch.

Review details

Best possible solution:

Review and, if correct, land the linked implementation with focused coverage for all-invalid exit-70/71 campaigns and archived runs lacking an explicit exit status.

Do we have a high-confidence way to reproduce the issue?

No; the report provides a specific native-campaign and aggregation path, but a current-main campaign was not run during this read-only review. The linked pull request's stated focused test run is useful supplemental evidence, not a reproduced issue result.

Is this the best way to solve the issue?

Yes, provisionally. Recording execution acceptance separately from diagnostic rewards and requiring an archived exit status is the narrowest path that preserves useful verifier data without treating invalid harness execution as benchmark success.

AGENTS.md: unclear because the file could not be read completely.

Remaining risk / open question:

• Until the linked pull request is reviewed and merged, fully invalid native-evaluation runs may still be included in score-eligible aggregates on current main.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Label changes

Label justifications:

• P2: Invalid benchmark acceptance can misstate aggregate scoring, but the report identifies a bounded native-evaluation and fleet-recovery repair path.
• impact:other: The affected surface is benchmark-run validity and score eligibility, which does not fit the more specific owned impact categories.

Evidence reviewed

What I checked:

• Concrete report and reproduction path: The report gives a focused current-main command path through python -m scripts.native_eval.aggregate and identifies the observable defect: all trials can exit with harness-specific code 71 while rewards still make the run eligible. (scripts/native_eval/aggregate.py, 884dd1bb5511)
• Established scoring invariant: Current repository documentation states that advisory/judge results cannot rescue a failed deterministic check, which supports treating fully invalid harness execution as ineligible rather than as completed score data. (README.md:282, 884dd1bb5511)
• Open implementation candidate: The linked pull request fix(native-eval): reject invalid harness runs #64 explicitly closes this issue and proposes structured execution outcomes, aggregate rejection, historical exit-70/71 handling, and removal of result-count-based fleet success inference. It remains open and unmerged. (9f4490e8e437)
• Repository history access limitation: Read-only shell inspection could not start because the sandbox failed before command execution (bwrap: loopback: Failed RTM_NEWADDR), so no independent local blame/log provenance was available for routing beyond the issue and linked implementation context.

Likely related people:

• vincentkoc: Reported the concrete native-evaluation acceptance failure and authored the open pull request that directly proposes the scoped fix. (role: reporter and linked implementation candidate; confidence: medium; commits: 9f4490e8e437; files: scripts/native_eval/aggregate.py)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.