perf(native-eval): install only the active harness on fleet leases

[vincentkoc]

What happened

An OpenClaw-only native fleet run installs OpenClaw, Codex, Claude Code, and Hermes on every fresh lease.

In a two-repetition OpenClaw run, both machines cloned and installed Hermes, synchronized its skills, and installed the unused Codex and Claude Code npm packages before dispatching the OpenClaw task.

Expected behaviour

Fleet bootstrap should install only the harness assigned to that lease. Shared runtime dependencies such as Docker, Node, uv, and LiteLLM should remain common.

Steps to reproduce

python -m scripts.native_eval.fleet \
  --run-index <openclaw-only-run-index.json> \
  --local-root <output-dir> \
  --runner-root . \
  --task-archive <tasks.tar.gz> \
  --env-file <provider.env> \
  --openclaw-package-tarball <openclaw.tgz>

Relevant output

Installing to /opt/shellbench-native/hermes-agent...
Installing dependencies...
Syncing bundled skills to ~/.hermes/skills/ ...

The same bootstrap also installs @openai/codex and @anthropic-ai/claude-code.

Environment

• Python version: 3.14
• OS: macOS controller, Ubuntu 26.04 leases
• ShellBench commit: 42ceda028876f5134697e4473ecfe691e0af3ca6

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 3, 2026, 5:04 AM ET / 09:04 UTC.

Summary
This issue remains necessary: current main still hydrates each fresh lease without passing its assigned harness into bootstrap, while the bootstrap script unconditionally installs OpenClaw, Codex, Claude Code, and Hermes. The report is source-reproducible, but it is MEMBER-authored and the toolchain-manifest contract for intentionally absent harnesses needs an explicit maintainer choice before a repair is promoted.

Reproducibility: yes. from source: a single-harness run reaches lease hydration without conveying its harness to bootstrap, and bootstrap unconditionally installs all four harnesses. A live lease run is still needed to measure the time and network reduction after the chosen manifest contract is implemented.

Maintainer decision needed

Question	Recommendation
Should native fleet audit manifests represent uninstalled harnesses explicitly while leases install only their assigned harness?	Adopt selected-harness manifests: Install only the assigned harness and make the manifest state which harnesses were installed, with absent tools represented explicitly rather than probed.

Why: The source-proven performance repair is narrow, but the current manifest executes version probes for all four tools and acts as retained audit evidence; maintainers should set the durable absent-tool representation before the schema and bootstrap behavior change together.

Next step
The implementation path is source-proven, but a maintainer must first approve the audit-manifest contract for tools that will no longer be installed on every lease.

Review details

Best possible solution:

Pass the assigned harness from the fleet controller into bootstrap, install only that harness plus shared dependencies, and record an explicit installed-harness set in the toolchain manifest with focused fleet and bootstrap regression coverage.

Do we have a high-confidence way to reproduce the issue?

Yes, from source: a single-harness run reaches lease hydration without conveying its harness to bootstrap, and bootstrap unconditionally installs all four harnesses. A live lease run is still needed to measure the time and network reduction after the chosen manifest contract is implemented.

Is this the best way to solve the issue?

Unclear until maintainers choose the absent-harness audit representation. Once chosen, passing the existing run harness into bootstrap and conditionally installing it is the narrowest maintainable repair.

AGENTS.md: not found in the target repository.

Remaining risk / open question:

• Selective installation must replace unconditional version probes in the toolchain manifest with an explicit representation of installed and intentionally absent harnesses; otherwise bootstrap can fail or audit artifacts can misleadingly imply every harness was available.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Label changes

Label justifications:

• P2: Unconditional installation wastes fleet lease time and network work, but it does not currently prevent benchmark execution.
• impact:other: The concrete impact is native evaluation setup overhead and cost rather than data, delivery, session, auth, or availability failure.

Evidence reviewed

What I checked:

• Current bootstrap lacks an active-harness input: Lease hydration invokes the bootstrap script after extracting the runner, but does not pass the run's harness; the harness is only supplied later when dispatching the benchmark. (scripts/native_eval/fleet.py:871, 884dd1bb5511)
• All harness installers remain unconditional on current main: The bootstrap script installs all three npm harnesses, installs Hermes separately, and calls every installer unconditionally before writing its manifest. (scripts/native_eval/bootstrap_beast.sh:83, 884dd1bb5511)
• Current harness selection occurs after bootstrap: The fleet dispatch path receives run.harness, confirming that the planned harness is available to the controller but is not yet used to select bootstrap dependencies. (scripts/native_eval/fleet.py:997, 884dd1bb5511)
• Behavior dates to the native matrix runner introduction: Blame attributes the unconditional npm, Hermes, and manifest logic to the commit that introduced the auditable native matrix runner; no later current-main change shown by the inspected history alters that bootstrap selection model. (scripts/native_eval/bootstrap_beast.sh:83, 69f75c6629c4)
• No stable-maturity classification is supportable: The required maturity shortlist helper could not locate qa/maturity-scores.yaml in this checkout, so no M4/M5 owner-surface label is applied.

Likely related people:

• Vincent Koc: Git blame attributes the unconditional harness installations and manifest-writing logic to the auditable native matrix runner introduction, and subsequent native-eval history also shows recent work in this area. (role: introduced native fleet bootstrap behavior; confidence: high; commits: 69f75c6629c4, 884dd1bb5511; files: scripts/native_eval/bootstrap_beast.sh, scripts/native_eval/fleet.py)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.