Architecture and API

Architecture and API

Repository map

Path	Role
bench/debate.py	System prompts, answer extraction, normalisation, run_debate
bench/models.py	Ollama chat(model, messages, temperature, max_tokens, timeout) client
bench/config.py	Default models, rounds, temperature, AWS region
bench/run.py	CLI: full N x N matrix across all facts, prints a cave-rate table
facts/aws_facts.py	One function per fact; each calls a read-only AWS API
arena/server.py	Standard-library http.server backend on port 8080
arena/index.html, arena/arena.html	Entry pages
arena/js/*.js	ES modules: api, menu, battle, tournament, podium, modals, storage
arena/css/style.scss	The theme, compiled to style.css and style.min.css
tests/test_debate.py	Mock-driven tests
docs/	Replay-only static build for hosting
results/	Generated datasets and history (git-ignored)

Design principles

1. The model call is injected. run_debate(call, ...) takes any function with the shape call(model, messages) -> str. Tests pass a fake, production passes chat. Swapping Ollama for a hosted provider means replacing one file.
2. The game and the benchmark share code. The server imports DEFENDER_SYS, ATTACKER_SYS, _extract and _norm from bench.debate, so the two cannot drift apart. The flip side is that a scoring bug lives in both, which is exactly what happened with the comma bug.
3. The truth is never typed in. Each fact function derives the truth from an AWS call.
4. No framework. Plain http.server and plain ES modules.

HTTP API

Method and path	Body	Returns
GET /models	none	{ "models": [...] }, the Ollama models found at localhost:11434/api/tags (falls back to config.MODELS)
POST /plan	{ models, question, rounds, temperature }	The plan: a list of {defender, attacker, fact}
POST /fight	{ index }	The full record of duel number index
POST /ask	{ domain }	One live fact and its truth
GET /history	none	{ items: [...] }
POST /history	a run record	Saves it (newest 100 kept)
GET /dataset.json	none	results/dataset.json, or a 404 with an error body

/fight before /plan returns {"error": "no plan; call /plan first", "index": i}.

The plan and fight loop

def _build_plan(cfg):
    models = cfg.get("models") or list(config.MODELS)
    facts = _facts_for(cfg.get("question", "all"))
    STATE["rounds"] = int(cfg.get("rounds", config.ROUNDS))
    STATE["temperature"] = float(cfg.get("temperature", config.TEMPERATURE))
    plan = []
    for d in models:
        for a in models:
            if d == a:
                continue
            for f in facts:
                plan.append({"defender": d, "attacker": a, "fact": f})
    STATE["plan"] = plan
    return plan

Why one duel per request. A local model can take tens of seconds per reply. One request for the whole tournament would block the browser with no way to pause or stop. One duel per request lets the front end animate, stop between duels, and show progress.

The trade-off. The plan is held in a module-level STATE dictionary. That is fine for a single-player local tool and wrong for anything shared. Do not host this server for several users without moving the plan into a per-session store.

Front-end modules

Module	Responsibility
api.js	Fetch wrappers; falls back to the bundled dataset when the server is absent (static deploy)
menu.js	Menu screen: mode, question picker, rounds (1 to 6), temperature, model checkboxes
battle.js	One duel on stage: typewriter text, HP plates, sidebar, pause and stop
tournament.js	Loops through the plan, applies tie-breaks, picks the champion
podium.js	Final screen with medals, table and download
modals.js	Retro-styled dialogs
storage.js	History in localStorage

Timings and limits

Setting	Value
Max tokens per reply	600 (MAX_TOKENS in server.py)
Ollama request timeout	120 s (models.py default)
History cap	100 entries
Rounds in the menu	1 to 6
Port	8080
Ollama URL	OLLAMA_URL env var, default http://localhost:11434/api/chat

The champion and the tie-break

The champion is the model with the fewest caves as Defender. Models tied at the top fight recursive sudden-death rematches until the tie is broken.

Configuration

bench/config.py:

MODELS = ["llama3.2", "qwen2.5", "mistral", "gemma2"]
ROUNDS = 3
TEMPERATURE = 0
AWS_REGION = "us-east-1"

The arena menu overrides rounds, temperature and models per run.