fix(ci): docs OpenHands eval — single trigger per ready PR head

[rickylabs]

Summary

Stops the docs OpenHands evaluation loop by dispatching only when an eligible PR becomes ready for review or receives an authorized /docs-eval rerun request. Claims each head SHA once, preserves docs-eval:skip and ci:full, and repairs the downstream LiteLLM startup failure observed in live logs.

Scope

• Archetype / area: GitHub Actions tooling
• Refs epic: Desktop Frontend — the full frontend as a native desktop app, the NetScript way #840
• Refs ci(docs): auto OpenHands minimax-M3 accuracy eval for docs-labeled PRs (skippable) #806 lineage

Slices

• S1 Prove ready-only, per-head exactly-once dispatch and repair the OpenHands runner — e0a22d36

Definition of Done

• Draft pushes and label churn cannot dispatch docs eval.
• Ready-for-review and authorized explicit request are the only dispatch paths.
• A durable head-SHA marker prevents a second trigger after success or failure.
• docs-eval:skip and ci:full remain explicit escape hatches.
• Focused event-matrix tests and YAML parsing pass.
• Opposite-family supervisor review/eval passes.
• Required CI is green.

Validation

• deno test --no-lock --allow-read .llm/tools/agentic/openhands/docs-eval-workflow_test.ts — PASS, 4/4
• YAML parse with jsr:@std/yaml — PASS, 2/2 workflows
• volatile-config guard — PASS, 4/4
• focused deno fmt --check — PASS
• git diff --check — PASS
• actionlint — unavailable; compensated by YAML parse and structural event-matrix assertions

Harness

• Run dir: .llm/runs/beta11-cli--orchestrator/slices/docs-eval-fix/
• Phase: impl — supervisor owns opposite-family review and formal evaluation.

Drift / Debt

• none
• No evaluator was self-dispatched. Do not merge until CI and supervisor-triggered evaluation pass.

[rickylabs]

[PHASE: IMPL]

Single-trigger docs-eval fix landed in e0a22d36 and was pushed with the explicit refspec HEAD:fix/docs-eval-loop.

Diagnosis

• docs(mcp): public-facing README rewrite + docs-site MCP coverage #858: 1 trigger / 1 failed OpenHands summary.
• docs: package READMEs to the public-introduction standard (batch B1 flagships) #861: 14 head-SHA triggers / 14 failed summaries during iteration.
• docs: main repository README rewrite (#816 pipeline) #862: 18 head-SHA triggers / 17 completed summaries at evidence capture.
• Representative raw OpenHands logs fail before the model call with ModuleNotFoundError: No module named 'fastapi'; bootstrap, OpenRouter provider selection, and Minimax M3 model selection had succeeded.

Implemented contract

• Dispatch only on ready_for_review or an exact authorized /docs-eval rerun PR comment.
• Never dispatch on open, synchronize/push, or label churn.
• Serialize per PR and permanently claim each head SHA with <!-- docs-openhands-eval head=<sha> -->; later success/failure summaries never reopen the claim.
• Preserve docs-eval:skip, ci:full, PAT-only chaining, trusted-base prompt loading, and the open-model guard.
• Install FastAPI explicitly in the OpenHands SDK environment, fixing the observed LiteLLM 1.92 MCP import crash.

Gate evidence

• deno test --no-lock --allow-read .llm/tools/agentic/openhands/docs-eval-workflow_test.ts — PASS, 4/4
• both edited workflow files parsed with jsr:@std/yaml — PASS, 2/2
• volatile-config guard — PASS, 4/4
• focused deno fmt --check — PASS
• git diff --check — PASS
• actionlint — unavailable; YAML parse and focused structural/event-matrix assertions provide the local syntax/policy proof.

Next

• Supervisor performs opposite-family review/evaluation and watches CI.
• No evaluator was self-dispatched. No merge, release action, or milestone closure was performed.

[rickylabs]

Supervisor opposite-family review — PASS (Fable 5, session 86d308d5)

Reviewed e0a22d36 (head fa7d0be7, delta is worklog-only).

Root cause confirmed: the previous workflow dispatched on opened/synchronize/labeled and its dedupe keyed on an identical unanswered trigger body — once OpenHands answered, every subsequent event reposted a fresh trigger, producing the endless loop.

Fix verified:

• Dispatch paths reduced to ready_for_review + authorized /docs-eval rerun comment (OWNER/MEMBER/COLLABORATOR only), serialized per-PR via concurrency group.
• Durable <!-- docs-openhands-eval head=<sha> --> marker claims each head SHA exactly once, regardless of eval outcome; a new commit (new head) is the re-eval path.
• docs-eval:skip honored on both event paths; ci:full opt-in preserved.
• Open-model guard (Minimax M3 only, closed-model strings rejected) and trusted-base-SHA prompt read preserved inside the claim script.
• openhands-agent.yml fastapi install matches the fix(ci): install fastapi for OpenHands agent runs #866 lineage fix.

Accepted limitation (noted, not blocking): a docs PR opened directly in ready state never emits ready_for_review, so it won't auto-dispatch — /docs-eval rerun is the documented path for that case.

Evidence: focused tests 4/4 PASS, YAML parse 2/2, fmt/diff-check PASS (gate table in the IMPL comment). Merging on CI green under the standing beta-11 authorization.