fix(native-eval): rebootstrap replacement leases

[vincentkoc]

What does this PR do?

Rehydrates replacement native-eval leases before dispatch.

Why?

The controller previously treated an old bootstrap timestamp as proof that a
new lease contained the runner. Recovery could then fail before the harness
started, producing a false benchmark failure.

Fixes #55

Changes

• record the exact bootstrapped_lease_id
• reuse bootstrap state only when it matches the active lease
• add a regression test for recovery with an old timestamp and replacement lease

Live proof

Recovered a legacy OpenClaw run whose manifest had an old bootstrap timestamp
but no bootstrapped_lease_id.

• rebound the run to its active replacement lease
• performed the full native-runner bootstrap on that lease
• persisted the exact active lease as bootstrapped_lease_id
• dispatched the OpenClaw harness successfully
• exported a verified final artifact with 4/4 completed results
• run exit code: 0

This is the exact recovery state the regression test covers; the controller no
longer trusts timestamp-only bootstrap state.

Tests

• 40 focused fleet tests pass
• Python 3.11 and Python 3.12 CI pass
• Ruff passes on the touched files
• fresh Codex autoreview reports no actionable findings
• live replacement-lease recovery completes a four-task OpenClaw run

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 3, 2026, 5:05 AM ET / 09:05 UTC.

ClawSweeper review

What this changes

The PR ties native-evaluation bootstrap reuse to the active Crabbox lease ID and adds a regression test for recovery after a lease replacement.

Merge readiness

⚠️ Ready for maintainer review - 2 items remain

This member-authored PR remains necessary: current main still uses a timestamp alone to decide whether a recovered native-evaluation lease is bootstrapped, while the branch records and checks the actual lease identity. The targeted regression test covers the reported replacement-lease recovery path; no discrete patch defect was found, but the intended legacy-manifest rehydration behavior warrants normal maintainer review before merge.

Priority: P2
Reviewed head: 8949dc8328e9b86a0cf1f80bcf3a976c67c4837a

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A focused repair with source-aligned behavior and a precise regression test; it needs ordinary maintainer confirmation of the intentional legacy recovery behavior.
Proof confidence	🌊 off-meta tidepool	Not applicable: This is a member-authored PR, so the external-contributor proof gate does not apply; the PR body nevertheless describes a successful live four-task recovery run with exit code 0.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: This is a member-authored PR, so the external-contributor proof gate does not apply; the PR body nevertheless describes a successful live four-task recovery run with exit code 0.
Evidence reviewed	5 items	Current-main behavior: Current main checks only bootstrapped_at_utc before dispatching a missing remote run, so a timestamp carried from an expired lease can suppress hydration of a replacement machine. Patch behavior: The branch reloads the persisted run after lease assignment, hydrates when bootstrapped_lease_id differs from the active lease, and persists that lease ID only after hydration succeeds. Regression coverage: The added test starts a recovery-required run with an old timestamp and replacement lease, verifies hydration occurred, verifies the active lease ID was persisted, and verifies dispatch completed.
Findings	None	None.
Security	None	None.

How this fits together

The native-evaluation fleet controller resumes benchmark runs from a persistent run index and dispatches them onto Crabbox leases. Before dispatching to a machine with no remote run state, it decides whether to hydrate the runner, task archive, and provider environment; this PR makes that decision lease-specific.

flowchart LR
  A[Persistent run index] --> B[Recovery controller]
  B --> C[Active Crabbox lease]
  C --> D[Remote run-state probe]
  D --> E{Bootstrap lease matches?}
  E -->|No| F[Hydrate runner and tasks]
  E -->|Yes| G[Reuse hydrated lease]
  F --> H[Dispatch benchmark harness]
  G --> H

Loading

Before merge

• Resolve merge risk (P1) - On upgrade, a legacy manifest without bootstrapped_lease_id will rehydrate once whenever the remote run state is missing, even if its stored timestamp is present; that deliberately replaces the remote runner, public tasks, and provider environment before dispatch.
• Complete next step (P2) - A member-authored patch with no mechanical defect needs a maintainer to confirm the intentional upgrade behavior before merging.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch scope	32 added, 1 removed across 2 files	The implementation is limited to one recovery decision, one persisted field, and one focused regression test.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #55
Summary: This PR is the concrete candidate fix for the reported replacement-lease recovery defect tracked by the linked issue.

Members:

• canonical: Rebootstrap native evals when recovery replaces a lease #55 - The issue describes the same timestamp-only bootstrap reuse failure, and the PR body explicitly connects this branch to that recovery behavior.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge-risk options

Maintainer options:

1. Confirm the legacy recovery tradeoff (recommended)
   Approve the intentional one-time rehydration of legacy manifests without a recorded lease ID before merging the targeted recovery repair.
2. Pause for a different upgrade policy
   Keep the PR open only if maintainers want a compatibility mode that trusts legacy timestamps rather than conservatively rehydrating missing remote state.

Technical review

Best possible solution:

Adopt lease identity as the durable bootstrap invariant, preserving the conservative one-time hydration path for legacy manifests and the existing successful recovery behavior for leases that already match.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: a recovery_required entry with an old bootstrapped_at_utc, no recorded bootstrap lease ID, and a replacement active lease follows the affected dispatch path. The branch adds a focused regression test for that exact state, but this read-only review did not execute it.

Is this the best way to solve the issue?

Yes. Recording the identity of the lease that actually received bootstrap is the narrowest durable fix; a timestamp cannot establish that a replacement machine has the required runner and task contents.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

• P2: This is a bounded native-evaluation recovery defect that can produce false benchmark failures after a replacement lease.
• merge-risk: 🚨 compatibility: The persisted bootstrap decision changes for existing manifests that lack a lease ID, causing a conservative rehydration on recovery.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: This is a member-authored PR, so the external-contributor proof gate does not apply; the PR body nevertheless describes a successful live four-task recovery run with exit code 0.

Evidence

What I checked:

• Current-main behavior: Current main checks only bootstrapped_at_utc before dispatching a missing remote run, so a timestamp carried from an expired lease can suppress hydration of a replacement machine. (scripts/native_eval/fleet.py:566, 884dd1bb5511)
• Patch behavior: The branch reloads the persisted run after lease assignment, hydrates when bootstrapped_lease_id differs from the active lease, and persists that lease ID only after hydration succeeds. (scripts/native_eval/fleet.py:566, 8949dc8328e9)
• Regression coverage: The added test starts a recovery-required run with an old timestamp and replacement lease, verifies hydration occurred, verifies the active lease ID was persisted, and verifies dispatch completed. (tests/test_native_eval_fleet.py:1276, 8949dc8328e9)
• Feature history: The native fleet controller appears to have been introduced and subsequently stabilized by Vincent Koc in the available current-main history; this PR is a narrow follow-up on that path. (scripts/native_eval/fleet.py:541, b9acd9f7a010)
• Current-main comparison: The PR head is not an ancestor of current main, and current main has no later commits on either touched path after this branch point; the lease-ID safeguard is therefore not already implemented there. (scripts/native_eval/fleet.py:566, 8949dc8328e9)

Likely related people:

• vincentkoc: Available history shows Vincent Koc introduced the native matrix runner, stabilized its fleet behavior, and authored this focused recovery fix. (role: native-evaluation feature owner and recent area contributor; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 8949dc8328e9; files: scripts/native_eval/fleet.py, tests/test_native_eval_fleet.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Confirm that conservative rehydration of legacy manifests without a recorded lease ID is the desired upgrade behavior.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (21 earlier review cycles; latest 8 shown)

• reviewed 2026-08-01T22:59:14.951Z sha 8949dc8 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T13:17:00.915Z sha 8949dc8 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T19:14:34.796Z sha 8949dc8 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T20:37:22.956Z sha 8949dc8 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T22:18:20.095Z sha 8949dc8 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T00:19:32.496Z sha 8949dc8 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T01:48:36.800Z sha 8949dc8 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T04:09:32.245Z sha 8949dc8 :: needs maintainer review before merge. :: none