Benchmarks live

Benchmarks: live rooms

The live half of the deliberation benchmark: what happened when the simulated participants were replaced by real models. The deterministic arms measure the protocol. These runs measure only whether a room of language models can hold it, what a host owes its agents so that it can, and — on a scenario with a recorded answer — whether deliberation beats a poll of the same members answering alone.

Benchmarks: delegation reports a later family of live rounds, against the expertise arms.

Live agents

--agent-cmd swaps the simulated participants for a real agent CLI, one process per authorized turn. These runs used opencode 1.18.25 against OpenRouter:

cargo run --release -p tinyhivemind-hive --example bench -- \
  --agent-cmd "opencode run --pure -m openrouter/~openai/gpt-mini-latest" --agents 5

They assert nothing. The deterministic arms measure the protocol, and a handful of live episodes could not measure anything. What they establish is that real models hold the trace grammar, and what they surfaced is a set of host-side obligations the simulation cannot see. All four were fixed in the harness, and each fix is one a host owes its agents rather than something the library can impose:

what happened live	the fix
Models coined #rollout and #rollout-strategy for one idea; support split across two names never adds up to a quorum.	The prompt names the options already on the floor, folded through the library's own standings, with how many supporters each holds and how many it needs.
Four consecutive turns restated the same !question verbatim. repetition_cap damps a restated support and cannot see this.	The prompt shows a participant its own last line and asks for something that moves the room on.
Models wrote !commit while the room was still deliberating, which adds no supporter; one episode spent its whole budget recording a decision it never reached.	The prompt offers only the moves that count in the turn's phase, so !commit appears only under Phase::Commit.
Four of five proposals dropped the #, so the lines named no topic and deposited nothing.	The grammar's sigils are called out explicitly, with a right and a wrong example.

A three-member room also exhausted its budget before the policy fix above, because a threshold of three on a three-member desk is unanimity and one !object had silenced an advocate. That is the second bound on quorum, found live before it was measured.

After the fixes, a five-member room converged on #rollout in 6 turns and 18 seconds of wall clock, with every proposal well formed and one commit. A three-member room converged in 5 turns and 14 seconds.

A real problem, and a control that can lose

Everything above measures whether a model can hold the grammar. None of it measures the thing the protocol exists for, because the synthetic brief has no answer and every member is told the same nothing. --scenario adds a problem that does:

cargo run --release -p tinyhivemind-hive --example bench -- \
  --agent-cmd "opencode run --pure -m openrouter/openai/gpt-5-mini" \
  --scenario crates/tinyhivemind-hive/examples/bench/scenarios/checkout-503.txt \
  --repeat 5

A scenario file carries a shared brief, the options under the ids the room should use, a private brief per member, and the recorded answer. The private briefs never reach the shared journal. A fact every member can already read is not private information, and a room whose members all start from the same facts has nothing to pool — which is the quiet reason most multi-agent results are uninteresting. Both arms then run against the same real agents: a deliberation episode, and an independent poll of the same members answering alone.

The control has to be able to lose

Two scenario designs were thrown away before one separated the arms, and how they failed is the most portable finding here.

Both made the correct option #retries — a client retry storm after a release. That is the canonical 503 story, and a language model reaches for it unprompted. Every member solved it alone and the vote scored five out of five in both designs. An arm that cannot lose is not a control, and a scenario whose answer survives deleting every private brief is not measuring deliberation at all.

The third design is a hidden profile: the brief itself plants the retry storm, and the recorded answer is the boring one, #pool. Reaching it needs four facts held by four different members — the pool caps at 20 and a request waiting 250ms is failed with a 503; each in-flight request holds one connection for its whole life; in-flight requests have sat at 24–31 since 09:14 with traffic flat; the release quintupled how long each request lasts — plus a fifth that kills the decoy, that the retry path shipped disabled and has never fired. No member holds two of the four, and any one alone is inert.

What the rooms did

Five members, quorum 3, budget 15, one episode per row:

agent CLI	hive	turns	vote plurality
claude -p --model sonnet	#pool, correct	10	#retries, wrong (3/5)
opencode → openai/gpt-5-mini	#pool, correct	8	#retries, wrong (4/5)
opencode → openai/gpt-5-mini	#pool, correct	8	#retries, wrong (3/5)
opencode → qwen3-235b-a22b	#pool, correct	11	tied 2–2–1, no answer

Four rooms reached an answer no member could reach alone, at 8–11 turns of a 15-turn budget and 45–75 seconds of wall clock. The poll never once returned the recorded answer.

Ten further episodes on one model, gpt-5-mini, put a rate on that:

arm            hive correct   turns/ep   vote correct
deliberation        6/10          7.6          —
independent poll      —            —          0/10

The poll lost all ten, always to #retries. Across every episode ever run on this scenario it has returned the recorded answer zero times out of sixteen, which is the point of a hidden profile rather than a surprise: a member voting alone has its own facts and the brief, and both point at the decoy.

Deliberation is not reliable here either. Six in ten is what a room of this model achieves at this budget, and the four failures all fail the same way, described below. The honest summary is that the shape separates the arms cleanly and reproducibly — not that deliberation solves it.

Support is counted; grounds are not weighed

This is the finding worth acting on, and it comes from the rooms that got it wrong.

In all four failed episodes of the ten the decisive refutation — the retry path shipped disabled, zero retries fired today — was already in the transcript, at a sequence every later message could cite. It changed nothing. Members went on depositing !support #retries ^6, and three grounded supporters carried the decoy. A room fails this task in exactly one way, and it is this one.

The library counts distinct grounded supporters. It does not, and as a pure fold cannot, check that a cited message supports the claim it is cited for. So far so correct. But the grammar gives a member no way to say the thing that would have mattered: this evidence refutes that topic. !object >N ^M silences one advocate message. Killing a hypothesis with a fact means objecting to every advocate separately, one turn each, and a room runs out of budget before it runs out of advocates.

A negative evidence-to-topic link — a !refute #topic ^N that debits or caps a topic's standing rather than silencing one author — is the missing move. It is a fold over the same traces and needs no port. Whether it should gate quorum or only weight it is a design question and wants an ADR before an implementation.

Two more things the live rooms showed

A member's fact can attach to the wrong hypothesis. The auditor's concurrency evidence is compatible with both the true cause and the decoy. In the failed rooms the auditor deposited it and proposed #retries from it, and every later member cited that message as grounds for the decoy. Pooled information does not help if it lands under the wrong heading, and nothing in the protocol notices.

Cross-inhibition fires, and it can fire against the truth. The one live !object observed across every run was aimed at #pool — the correct option — arguing that raising the pool masks a duration problem. The mechanism worked exactly as specified and removed a supporter from the right answer. It is not wrong; it is neutral, and a benchmark that reports only its wins is not reporting it.

Two harness defects the runs exposed

gpt-5-mini copied the prompt's <one sentence> placeholders into its output verbatim, angle brackets and all, where they survived into every later citation of that message. claude -p produced the other one: !support #retries ^5 … points at #pool instead, a support marker on one topic whose prose argues for another. The library counted a supporter the author plainly did not intend.

Both prompt defects were repaired, and the repair was then run against the prompt it replaced: five episodes each, same model, same scenario. The arms tied at three correct out of five, 7.8 turns against 7.4. The change is therefore not detectably better or worse, and it is kept for the transcripts it produces rather than for any accuracy claim. An earlier four-correct-then-two- wrong split had suggested a regression; at ten episodes it was noise.

The vote control had a defect of its own. The qwen room polled two, two and one, and the harness reported a #pool plurality because the tally sorted stably and #pool was inserted first. A tie is not a decision, and resolving it by arrival order hands the control a win it did not earn. plurality now returns nothing on a tie.

Reproducing

cargo run --release -p tinyhivemind-hive --example bench -- \
  --agent-cmd "opencode run --pure -m openrouter/openai/gpt-5-mini" --agents 5
cargo run --release -p tinyhivemind-hive --example bench -- \
  --agent-cmd "opencode run --pure -m openrouter/openai/gpt-5-mini" \
  --scenario crates/tinyhivemind-hive/examples/bench/scenarios/checkout-503.txt \
  --repeat 5

Live mode is not in CI and needs a configured agent CLI. The full write-up and the transcripts are in docs/experiments/2026-09-01-live-hidden-profile.md.