Skip to content

Panel Mode

Peter Pak edited this page Jul 15, 2026 · 4 revisions

Panel Mode (Harness Arena)

Design agreed 2026-07-15 — not yet built. Run all enabled harnesses in parallel on the same prompt, present anonymized candidate responses, let the user select one; the selection is both the conversation's next step and a preference label for the thesis.

Why

The between-build comparison (swap harness per build) yields unpaired observations confounded by build-to-build differences. Panel mode yields paired comparisons — four harnesses answering the identical prompt against identical printer/database state — so "which harness earns user validation" becomes a statistics question (Bradley-Terry/Elo over selections) instead of an anecdote. Complements the objective metrics already recorded (tool calls, tokens, latency, agent_actions).

Decisions

  1. The broker owns the conversation state, not the harnesses. One canonical transcript per panel conversation (user turns + selected answers). Every turn, the full canonical transcript is re-sent to each enabled harness as a fresh headless one-shot — no native --resume in panel mode. Harnesses are stateless response generators. (This also sidesteps the agy -c cross-talk bug.) v1 renders the transcript with a simple role-tagged text format; a syntax closer to the harnesses' native formats is a later refinement.
  2. Selection = exactly one response. Only the chosen response enters the canonical transcript; the others are archived as candidates. If two harnesses give equivalent answers the user still picks one. No parallel execution ever: panel mode is analyst-preset, propose-only — if the selected response recommends a printer action, execution happens separately through the single-harness operator flow (or manually).
  3. Blinded selection. Candidates render as anonymized panes (A/B/C/D), shuffled server-side each turn; the harness identity is revealed only after the selection is persisted. The shown order is recorded so position bias can be checked. "Tie" / "none are good" outcomes are recordable — forced choice corrupts the signal.
  4. Harness↔model correlation is accepted and documented. Claude Code runs Claude models, Codex OpenAI models, Antigravity Gemini models — selections measure the stack. The model field is recorded on every candidate response for future reference. OpenCode is the deliberate exception: it is the multi-model harness (hosted models and the planned domain-adapted fine-tune), and OpenCode×{models} remains the model-isolating axis of the thesis.

Flow

user message ──▶ broker appends to canonical transcript
                 ├─▶ claude   (fresh one-shot, canonical context) ──┐
                 ├─▶ opencode (same)                                ├─ SSE per pane,
                 ├─▶ codex    (same)                                │  shuffled A–D
                 └─▶ agy      (same)                                ┘
user selects one (or tie/none) ──▶ agent_selections row
                                ──▶ chosen text appended to canonical transcript
                                ──▶ harness identities revealed in GUI
next turn repeats with the updated canonical transcript

Toggles: each panel has an enabled_harnesses set — disable two or three to save tokens; a panel of one degrades to today's single-chat behavior. Panel mode is opt-in per conversation (4× token cost).

Data model (sketch)

  • Canonical conversation: an agent_conversations row (role='panel'); its agent_messages hold user turns + selected answers only.
  • Candidate responses: one agent_sessions row each (role='panel:candidate', existing columns cover harness/model/tokens/latency/transcript), linked to the panel conversation + turn.
  • agent_selections: panel conversation id, turn, candidate session ids, chosen session id (null for none), outcome (chosen|tie|none), shown order, optional reason string, decided_at. Designed so Bradley-Terry fits directly from this table.

Prerequisites (from the harness review, 2026-07-15)

Must land before arena data collection:

  • spawn error handlers + per-turn timeouts in the broker (one hung CLI must show a timeout card, not brick the panel).
  • Permission-envelope fix or at minimum an enforcement-level column — analyst is hard-enforced only for Claude today; selections must not measure who had the longer leash.
  • Model capture for opencode/agy (currently NULL unless --model passed).

Strongly recommended: server-side logging of ALL MCP tool calls in the plugin (not just *_set) — the only harness-independent way to compare tool behavior, since opencode/agy transcripts are plain text.

Evaluation plan

  • Primary: selection outcomes → Bradley-Terry/Elo with confidence intervals; report ties/none rates.
  • Bias checks: position effects from shown-order; selection reasons rubric (author is the sole evaluator — define the rubric up front).
  • Secondary: per-candidate latency, tokens, tool-call counts, and objective correctness on database-grounded questions.

Clone this wiki locally