Repository navigation
Panel Mode
Design agreed 2026-07-15 — not yet built. Run all enabled harnesses in parallel on the same prompt, present anonymized candidate responses, let the user select one; the selection is both the conversation's next step and a preference label for the thesis.
The between-build comparison (swap harness per build) yields unpaired
observations confounded by build-to-build differences. Panel mode yields
paired comparisons — four harnesses answering the identical prompt
against identical printer/database state — so "which harness earns user
validation" becomes a statistics question (Bradley-Terry/Elo over selections)
instead of an anecdote. Complements the objective metrics already recorded
(tool calls, tokens, latency, agent_actions).
-
The broker owns the conversation state, not the harnesses. One
canonical transcript per panel conversation (user turns + selected
answers). Every turn, the full canonical transcript is re-sent to each
enabled harness as a fresh headless one-shot — no native
--resumein panel mode. Harnesses are stateless response generators. (This also sidesteps the agy-ccross-talk bug.) v1 renders the transcript with a simple role-tagged text format; a syntax closer to the harnesses' native formats is a later refinement. - Selection = exactly one response. Only the chosen response enters the canonical transcript; the others are archived as candidates. If two harnesses give equivalent answers the user still picks one. No parallel execution ever: panel mode is analyst-preset, propose-only — if the selected response recommends a printer action, execution happens separately through the single-harness operator flow (or manually).
- Blinded selection. Candidates render as anonymized panes (A/B/C/D), shuffled server-side each turn; the harness identity is revealed only after the selection is persisted. The shown order is recorded so position bias can be checked. "Tie" / "none are good" outcomes are recordable — forced choice corrupts the signal.
-
Harness↔model correlation is accepted and documented. Claude Code
runs Claude models, Codex OpenAI models, Antigravity Gemini models —
selections measure the stack. The
modelfield is recorded on every candidate response for future reference. OpenCode is the deliberate exception: it is the multi-model harness (hosted models and the planned domain-adapted fine-tune), and OpenCode×{models} remains the model-isolating axis of the thesis.
user message ──▶ broker appends to canonical transcript
├─▶ claude (fresh one-shot, canonical context) ──┐
├─▶ opencode (same) ├─ SSE per pane,
├─▶ codex (same) │ shuffled A–D
└─▶ agy (same) ┘
user selects one (or tie/none) ──▶ agent_selections row
──▶ chosen text appended to canonical transcript
──▶ harness identities revealed in GUI
next turn repeats with the updated canonical transcript
Toggles: each panel has an enabled_harnesses set — disable two or three to
save tokens; a panel of one degrades to today's single-chat behavior. Panel
mode is opt-in per conversation (4× token cost).
- Canonical conversation: an
agent_conversationsrow (role='panel'); itsagent_messageshold user turns + selected answers only. - Candidate responses: one
agent_sessionsrow each (role='panel:candidate', existing columns cover harness/model/tokens/latency/transcript), linked to the panel conversation + turn. -
agent_selections: panel conversation id, turn, candidate session ids, chosen session id (null for none), outcome (chosen|tie|none), shown order, optional reason string, decided_at. Designed so Bradley-Terry fits directly from this table.
Must land before arena data collection:
-
spawnerror handlers + per-turn timeouts in the broker (one hung CLI must show a timeout card, not brick the panel). - Permission-envelope fix or at minimum an enforcement-level column — analyst is hard-enforced only for Claude today; selections must not measure who had the longer leash.
- Model capture for opencode/agy (currently NULL unless
--modelpassed).
Strongly recommended: server-side logging of ALL MCP tool calls in the
plugin (not just *_set) — the only harness-independent way to compare tool
behavior, since opencode/agy transcripts are plain text.
- Primary: selection outcomes → Bradley-Terry/Elo with confidence intervals; report ties/none rates.
- Bias checks: position effects from shown-order; selection reasons rubric (author is the sole evaluator — define the rubric up front).
- Secondary: per-candidate latency, tokens, tool-call counts, and objective correctness on database-grounded questions.