Phase 3 (commits 5-13): runner registry full lifecycle — dispatcher, callbacks, dashboard, runner SDK, CLI, e2e, docs, version bump

[risjai]

Summary

Closes the loop on Phase 3 — the dashboard becomes a control surface instead of a viewer. Operators register long-lived runner processes; clicking "Run replay" on a session step HMAC-signs a webhook to the runner; the runner executes the agent under a replay context; progress streams back to the dashboard live via WebSocket.

This is commits 5-13 of the 13-commit Phase 3 plan, continuing from commits 1-4 already merged in PRs #150, #151, #152, #153.

What ships

commit	What
5	Webhook dispatcher (HMAC signing, 5s timeout) + lease reaper (30s scan)
6	POST /api/sessions/{sid}/replay-jobs (dual-shape A/B) + POST /api/replay-jobs/{id}/events (X-Rewind-Runner-Auth, ownership check, atomic event ingestion) + WS broadcast
7	Web RunnersPage (list / register / regenerate / remove + raw-token reveal modal)
8	Web RunReplayButton + ReplayJobModal + useReplayJob hook + replay_job_update WebSocket frame
9	Python rewind_agent.runner library (RunnerConfig, compute_signature, verify_signature, ProgressReporter, asgi_handler, @handle_replay) + ExplicitClient.attach_replay_context + env-var bootstrap
10	rewind runners {list, add, remove, regenerate-token} CLI
11	E2E test exercising the full lifecycle against a real binary + stub runner; REWIND_ALLOW_LOOPBACK_WEBHOOKS dev escape hatch
12	docs/runners.md operator how-to + decision matrix updated 3-way → 5-way (also bundles deferred Phase 2 doc)
13	Rust workspace 0.13.0 → 0.14.0 across 4 mirror files (Python SDK + MCP versions stay per CLAUDE.md track-2)

API / wire format

Outbound dispatch (Rewind → runner):
```
POST <runner.webhook_url>
X-Rewind-Job-Id:
X-Rewind-Signature: sha256= # HMAC-SHA256(raw_token, job_id || \n || raw_body)
Content-Type: application/json
{ "job_id", "session_id", "replay_context_id", "base_url" }
```

Inbound events (runner → Rewind, mounts OUTSIDE bearer middleware):
```
POST /api/replay-jobs/{id}/events
X-Rewind-Runner-Auth: # SHA-256 lookup; ownership check on job.runner_id
{ "event_type": "started|progress|completed|errored", "step_number?", "progress_total?", ... }
```

Reference vectors pinned both sides: crates/rewind-web/src/dispatcher.rs::compute_signature_matches_python_reference ↔ python/tests/test_runner.py::test_compute_signature_matches_rust_reference.

State machine + reaper

Same SQL-level terminal-state guard from #152 round 2 protects every transition. Reaper marks dispatched-but-never-started → errored stage='dispatch' and in_progress-without-heartbeat → errored stage='lease_expired'.

Bootstrap (server)

```bash
export REWIND_RUNNER_SECRET_KEY="$(openssl rand -base64 32)" # required
export REWIND_PUBLIC_URL="https://rewind.example\" # for non-local
rewind web
```

Unset key → 503 with bootstrap message. Set-but-malformed → server panics at startup (per #153 MEDIUM 5 fix).

Test plan

• cargo test --workspace — green
• cargo clippy on rewind-web --lib, rewind-store --all-targets, rewind-cli — clean (pre-existing test-file warnings on recording_api_tests / transcript_sync_tests unchanged from master)
• 24 runners_tests + 12 new replay_jobs_tests integration tests
• 481 pytest pass, 1 skipped (Python — includes 19 new test_runner.py + 1 e2e)
• web build green (426 KB / 119 KB gz; 1705 modules)
• E2E flow verified end-to-end via python/tests/e2e/test_runner_replay_flow.py
• All previous phase work intact (Phase 1 + 2 + 3 commits 1-4)

Notes for reviewers

• 9 logical commits (5-13). Each commit builds on its predecessor and is reviewable on its own.
• The REWIND_ALLOW_LOOPBACK_WEBHOOKS env escape is documented as DEV-ONLY in both the registration handler and docs/runners.md. Production deployments should never set it.
• cancellation is fire-and-forget in v1 (operator marks errored; runner not notified). Cooperative cancel deferred to v3.1 per the plan.
• polling mode is still rejected at registration (deferred to v3.1, per feat(rust): /api/runners CRUD + AES-256-GCM crypto module (Phase 3, commit 4/13) #153 HIGH 1).
• Token rotation + runner deletion still refuse while non-terminal jobs exist (per feat(rust): /api/runners CRUD + AES-256-GCM crypto module (Phase 3, commit 4/13) #153 HIGH 3 + MEDIUM 4).

Made with Cursor

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
rewind	Ready	Preview, Comment	Apr 28, 2026 7:03am

[risjai]

Santa Verdict: NAUGHTY

Both independent Santa reviewers returned FAIL. CI is green now, and this PR covers a lot of the Phase 3 lifecycle, but there are still blocking correctness/security issues in the runner replay path.

Blocking Findings

• Shape A accepts strict_match but does not apply it. CreateReplayJobShapeA includes strict_match, but after create_replay_context(...) the code never calls set_replay_context_strict_match. The dashboard can request strict replay and still get warn-on-divergence behavior. Add the setter and an integration test proving strict runner replay returns/raises on divergence.

• Runner replay does not bind live recordings to the fork timeline. ExplicitClient.attach_replay_context() only sets _session_id and _replay_context_id; it does not set _timeline_id. The dispatch payload also omits fork_timeline_id, and the SDK does not resolve the replay context's timeline before running the agent. Cache misses after the fork can therefore record new live steps into the root timeline instead of the replay fork. Include the fork/context timeline in the payload or make attach_replay_context() fetch context metadata and set _timeline_id before any live miss can record.

• Shape B reuse does not validate cursor state or context expiry. It checks session ownership and in-flight jobs, but does not reject replay contexts whose cursor was already advanced (current_step != from_step) or expired/near-expired contexts. A reused context can start from the wrong ordinal. Add current-step and TTL validation as described in the plan.

• Runner SDK ignores the dispatch payload base_url for callbacks. The payload includes base_url, but ProgressReporter builds event URLs from RunnerConfig.rewind_base_url / REWIND_URL. Dispatches only report progress correctly if the runner separately has matching config. Default the reporter to payload.base_url, with config override only when explicitly requested.

• Outbound webhook HMACs are replayable. The signature covers job_id || body, but has no timestamp, delivery id, expiry window, or runner-side job-id/delivery dedupe before scheduling user code. A captured signed webhook can be replayed to launch the agent again even if later event callbacks are rejected. Add X-Rewind-Timestamp + tolerance window and runner-side idempotency keyed by job_id or delivery_id before invoking the handler.

• SSRF validation is registration-time only and is vulnerable to DNS rebinding. validate_export_endpoint() is called when the runner URL is registered, but Dispatcher later uses reqwest against the stored URL and explicitly does not revalidate. If DNS changes after registration, the dispatcher can POST to private/link-local/metadata IPs. Revalidate/pin resolution at dispatch time, or use a connector/resolver that rejects private/reserved targets on connect. Strongly consider requiring HTTPS for non-loopback webhooks.

• Dashboard progress can miss fast job updates. The hook/modal subscribes after job creation, and the WebSocket callback setup can miss transient ReplayJobUpdated broadcasts for fast jobs that complete before the modal is listening for the returned job_id. The UI should fetch current job state after dispatch and/or subscribe before dispatch with a correlation token; WebSocket events should be treated as invalidation, not the only source of truth.

Also Noted

• The implementation reintroduces cancellation via API/UI despite the revised plan explicitly deferring cancellation until a cooperative runner protocol exists. Remove it for v1 or make it clear that this is only server-side abandonment and cannot stop the runner/agent.
• The E2E path uses a custom stub runner that reads payload.base_url directly, so it does not exercise the shipped rewind_agent.runner SDK behavior where the callback URL bug lives.

Passes

• CI is green.
• Versioning looks consistent with the repo rules.
• Runner token registration/regeneration and basic callback ownership checks are covered better than the earlier PRs.

Recommended Fix Set

• Wire strict_match into Shape A replay context creation.
• Ensure runner replay sets the fork timeline context before live misses can record.
• Validate Shape B current_step == from_step and context freshness/TTL.
• Make ProgressReporter use dispatch base_url by default.
• Add dispatch freshness/idempotency protections to HMAC webhooks.
• Revalidate/pin webhook destinations at dispatch time against SSRF policy.
• Add tests for strict Shape A, fork-timeline live miss, Shape B consumed/expired contexts, SDK callback base URL, HMAC replay prevention, DNS rebinding/dispatch-time SSRF, and fast UI completion updates.

[risjai]

All 7 blockers + 2 noted findings addressed in 27bdf09.

#	Severity	Finding → Fix
F1	BLOCKER	strict_match accepted but ignored → Shape A now calls set_replay_context_strict_match after creating the context. Tests: shape_a_strict_match_true_sets_replay_context_strict_flag, shape_a_strict_match_omitted_defaults_to_warn_only.
F2	BLOCKER	Live recordings not bound to fork timeline → dispatch payload now carries replay_context_timeline_id; Shape B response also returns it. Python attach_replay_context(session_id, replay_context_id, timeline_id=...) sets _timeline_id contextvar. Tolerant decode for back-compat.
F3	BLOCKER	Shape B doesn't validate cursor/TTL → refuses contexts whose current_step != from_step or older than 1h TTL. New Store::_test_set_replay_context_last_accessed helper for fast TTL-expiry tests. Tests: shape_b_rejects_context_whose_cursor_already_advanced, shape_b_rejects_expired_context.
F4	BLOCKER	ProgressReporter ignored payload base_url → constructor takes optional base_url; asgi_handler builds it from payload.base_url. Falls back to config.rewind_base_url for direct/test usage. Tests: test_progress_reporter_prefers_explicit_base_url_over_config, test_progress_reporter_falls_back_to_config_when_base_url_omitted.
F5	BLOCKER	HMAC webhooks replayable → signed input now HMAC-SHA256(token, timestamp \\n job_id \\n body). New X-Rewind-Timestamp header with ±5min tolerance enforced by SDK. Process-local seen-cache of recent job_ids defeats replays inside the tolerance window (200 + duplicate=true). Reference vectors re-pinned both sides. Tests: test_verify_signature_rejects_stale_timestamp, test_verify_signature_rejects_future_timestamp, test_asgi_handler_replay_attack_short_circuits_with_duplicate + matching Rust compute_signature_changes_when_timestamp_changes.
F6	BLOCKER	Registration-time SSRF only → Dispatcher::dispatch re-runs url_guard::validate_export_endpoint at dispatch time. Closes the DNS-rebind window between registration and dispatch. Residual race against reqwest's connect-time re-resolve documented as a known limitation requiring a custom hyper connector (deferred to v3.1).
F7	BLOCKER	UI misses fast updates → useReplayJob treats WebSocket frames as INVALIDATION (refetch GET /api/replay-jobs/{id}) rather than source-of-truth. After dispatch returns 202, immediately fetches current job state to seed UI. Out-of-order / missing frames both resolve to "latest server state".
N1	NOTED	Cancellation shouldn't have shipped (deferred to v3.1) → removed DELETE /api/replay-jobs/{id} endpoint, the dashboard cancel button, the API client method, and updated docs/runners.md to point operators at "kill the runner process; lease reaper marks errored".
N2	NOTED	E2E doesn't exercise SDK base_url logic → SDK code paths now have explicit unit-test coverage in test_runner.py (the F4 reporter tests above + replay-attack idempotency). The e2e was rewritten to verify the F2 (replay_context_timeline_id) + F5 (X-Rewind-Timestamp + 3-line signing recipe) wire-format additions; the raw stub stays for test-runtime stability (a dedicated event loop per HTTPServer request was flaky).

Test posture after the fix:

• 16 replay_jobs_tests pass (was 12, +4 for F1 + F3)
• 28 runner unit tests pass (was 19, +9 for F4 + F5 + F2 coverage)
• 108 rewind-web lib tests (adds 2 new dispatcher tests covering timestamp variation + 3-line signing input)
• 102 rewind-store tests (_test_set_replay_context_last_accessed helper exposed publicly with _test_ prefix + docstring contract)
• 490 pytest pass (1 skip), ruff clean, clippy clean on touched crates, web build green

Wire-format changes (operators take note):

• Outbound webhooks now carry X-Rewind-Timestamp and the signed input includes it. Pre-Phase 3 (commits 5-13): runner registry full lifecycle — dispatcher, callbacks, dashboard, runner SDK, CLI, e2e, docs, version bump #154 runners verifying with the 2-line input will see signature mismatches on dispatches from a v0.14.x server. The Python SDK falls back to the legacy 2-line input only when the timestamp header is absent (defensive — production dispatchers always emit it).
• Dispatch payload gains replay_context_timeline_id (not breaking; old runners ignore unknown fields).

CI will pick up the push. Ready for re-review.

[risjai]

Santa Re-Review: still NAUGHTY

Most of the previous blockers are addressed and CI is green, but I still see one remaining correctness blocker plus one important subprocess/bootstrap gap.

Remaining Blocker

• Runner callback race can still drop the started event and leave jobs stuck. The Python runner asgi_handler() schedules _run() before returning 202; _run() immediately calls reporter.started(). The server only transitions pending -> dispatched after the dispatcher receives the runner’s 202 and applies the outcome. If started arrives while the job is still pending, the state machine rejects it (Started is only legal from dispatched). ProgressReporter only logs the 409, so the handler continues and later progress/completed events can also be rejected or leave the job in dispatched until the reaper marks it errored.

Suggested fixes:

• Move job to dispatched before sending the webhook, then revert/mark errored if dispatch fails; or
• Allow authenticated started to transition pending -> in_progress for this dispatch race; or
• Change the runner SDK to send started only after a server-ack/state poll confirms dispatched.

Also Still Concerning

• Env-var bootstrap still doesn’t carry the timeline id. _install.py reads REWIND_SESSION_ID and REWIND_REPLAY_CONTEXT_ID, but not a replay context timeline id. So subprocess-based agents using env bootstrap can still miss _timeline_id, meaning live misses may record into the wrong timeline. The direct SDK path supports attach_replay_context(..., timeline_id=...), but the env path does not.

Please add REWIND_REPLAY_CONTEXT_TIMELINE_ID (or equivalent) to the dispatch/subprocess environment and pass it through to attach_replay_context() during bootstrap.

[risjai]

Both findings from the round-2 Santa pass addressed in f795f2b.

#	Severity	Finding → Fix
BLOCKER	Runner callback race drops started	create_replay_job now transitions pending → dispatched SYNCHRONOUSLY before tokio::spawn. The runner ALWAYS finds the job in dispatched when it tries to call back. apply_outcome restructured: success path only sets deadline+lease; failure path transitions dispatched → errored (legal startup-failure transition per #152 round 2). If started lands first (very fast runners), the failure-path transition becomes a no-op (SQL guard refuses Dispatched → Errored when current state is in_progress) and the run proceeds. Response state field flips from pending to dispatched accordingly — dashboards see the real state immediately.
CONCERNING	Env-var bootstrap missing timeline	_install._bootstrap_replay_context_from_env now reads REWIND_REPLAY_CONTEXT_TIMELINE_ID and forwards it to attach_replay_context(timeline_id=...). WARN log fires when the env var is missing (subprocess-bootstrap path matches the direct SDK path). Documented in docs/runners.md with example subprocess.run(env=...) snippet.

Picked option (a) from the suggested fix set ("Move job to dispatched before sending the webhook, then revert/mark errored if dispatch fails") — preserves the strict state machine from #152 round 2 without re-widening it. Options (b) and (c) would either weaken the state machine (b) or require runner-side polling (c, additional round-trip per dispatch).

Subtle property: if a very fast runner emits started AFTER the pre-spawn transition but BEFORE the dispatcher's HTTP call returns, the apply_outcome failure transition (Dispatched → Errored) becomes a no-op because the SQL guard WHERE state NOT IN ('completed','errored') matches but the current state is now in_progress, not dispatched — so the UPDATE matches in_progress rows too. Wait, that means a started-then-failed-dispatch combo could mark in_progress jobs errored. Let me re-verify the guard...

Actually the guard is WHERE id = ? AND state NOT IN ('completed', 'errored') — it matches pending, dispatched, or in_progress. So a failure-path Errored transition WOULD overwrite an in_progress job. That's a small follow-up to consider: tighten the failure-path guard to only target dispatched state. Filed for follow-up; non-blocking for this PR since the practical race window (between pre-spawn transition and apply_outcome) is the duration of the dispatch HTTP call, ~tens to hundreds of ms.

Test posture:

• 18 replay_jobs_tests pass (was 16, +2): create_replay_job_response_state_is_dispatched_not_pending and started_event_accepted_immediately_after_dispatch_response prove the race is closed.
• 30 runner unit tests pass (was 28, +1): new test_install_bootstraps_with_timeline_id_from_env covers the env-var path; existing test_install_bootstraps_from_env updated to assert _timeline_id stays None when the env var is absent.
• 491 pytest pass; clippy + ruff clean; web build green.

CI will pick up the push.

[risjai]

Self-caught follow-up to my previous comment, pushed as 023bd3f.

I called out the in_progress-clobber risk in my reply ("that means a started-then-failed-dispatch combo could mark in_progress jobs errored"), then realized I should fix it now rather than file as follow-up. The risk is real — pre-spawn transition + fast runner + failed dispatch HTTP call would have the failure path's advance_replay_job_state(Errored) match the in_progress row.

Fix: new Store::mark_dispatched_job_as_errored helper with the strict guard WHERE id = ? AND state = 'dispatched'. The failure-path apply_outcome uses this; UPDATE is a no-op when state has already advanced to in_progress, so the legitimate run continues.

New test failed_dispatch_does_not_clobber_already_in_progress_job inserts an in_progress row directly, calls the helper, and asserts the row is unchanged.

19 replay_jobs_tests pass (was 18, +1). Full suite remains green.