claque

Stress-test your conversational AI agent with a crowd of simulated users, and a judge you can actually trust.

_{A crowd of personas hammering an agent, graded live: solved, failed, or flagged unsafe.}

---

Your demo shows the happy path. claque shows you everything else.

A claque was a crowd hired to react in 19th-century Paris theaters. This one is hired to break your agent: it runs a cast of persona-driven simulated users against it (the confused elder, the terse expert, the refund seeker, the prompt injector), holds a full multi-turn conversation with each, and grades every one with a calibrated LLM judge. Then it tells you where your agent fails, with a confidence interval on every number.

Quickstart

pip install claque
claque demo --offline      # zero API calls, writes a graded HTML report

That runs eight scripted personas against a deliberately mediocre demo bot and opens a self-contained report, no credentials needed. Prefer uv? uv tool install claque. For live runs, add the anthropic extra and drop --offline (see the full quickstart). Point it at your own agent in one line:

uv run claque run --agent https://your-api.com/chat --personas ./personas

What you get

• A hostile crowd, not an echo chamber. Personas pin emotional state, literacy, patience, and a realistic way to give up, so a model playing "angry customer" does not thank your agent by turn three. Generate a stratified cast, or turn your own production logs into one.
• A judge you can audit. Anchored 0-3 rubrics as reviewable YAML, evidence quoted from the transcript, and claque calibrate: Cohen's kappa against your own blind labels. Every headline number carries a 95% interval and its sample size.
• Words vs. deeds. claque grades what the agent did, not just what it said. It catches false successes: the agent replies "your refund is processed" but never called the refund tool. A text-only grader cannot see that.
• Grade what already happened. claque analyze points the same calibrated judge at your agent's real past transcripts (a two-voice JSONL, or pulled from Intercom/Fin), clusters them by archetype, and reports each with a sample size and interval. Test before you ship, or measure what already shipped.
• Plug in any agent. A Python callable, an HTTP endpoint, an OpenAI-compatible server, MCP, A2A, submit-and-poll, or a Claude Agent SDK agent. The agent keeps its whole tool loop; claque only speaks user turns to it.
• Red-team, not just QA. An adversarial pack (prompt injection, secret extraction, a Crescendo attack, PII fishing) mapped to the OWASP LLM Top 10 and the Agentic Top 10, graded on a severity scale.
• Built for real agents. Per-turn timeouts, transient-error retries with backoff, rate limiting, --resume for a run that died halfway, and experienced-latency p50/p95, so you can point it at a production endpoint, not just a toy.
• Surfaces that scale. A live terminal arena you can watch, and a self-contained HTML report that stays readable from 8 to 1000 conversations (archetype rollup, failure clusters, latency).
• Close the loop. Export the failures to an SFT/DPO dataset, fine-tune, re-test, and prove the delta only counts when the intervals are disjoint.
• Offline-first. The whole test suite (1400+ tests) runs with no credentials, no network, no spend. Deterministic scripted mode is the CI default.

Point it at your agent

An agent is anything that maps a conversation to its next reply. The --agent spec covers the common shapes without glue:

Your agent is...	Spec
a deployed HTTP endpoint	https://your-api.com/chat
an OpenAI-compatible server (vLLM, Groq, Mistral...)	openai:your-model
Python code in your repo	your_pkg.agents:support_agent
a Claude Agent SDK agent	agent-sdk:config.yaml
MCP / A2A / async submit-and-poll	mcp:   a2a:   async:

The config file, when there is one, is the agent's address, not the agent. See docs/adapters.md.

Why you can trust the numbers

Most LLM judges are vibes with a number attached. claque is built the other way around: it is presumed guilty until calibrated.

• claque calibrate samples a run, serves you the transcripts in a local blind UI (the judge's score is hidden from you), and computes Cohen's kappa per criterion. It never writes a label itself.
• claque judge selftest proves the judge is not fooled by content-free "master key" replies (a bare : fools some judges 35% of the time).
• Until the judge is calibrated, the docs tell you to trust deltas, not absolute scores. Out loud.
• claque chart renders the headline rate, its confidence interval and the per-criterion means as one PNG. A number worth sharing is a number worth showing with its interval attached, so the picture carries the sample size and the interval method rather than a bare percentage.

pip install 'claque[chart]'
claque chart --out run.png --title "release gate"

Charting is an optional extra on purpose: claque installs with a single dependency, and a testing library should not pull a plotting stack onto every machine that runs it. The image is designed to stay readable when its long edge is scaled to 448 pixels, which is roughly what a phone or an image-ingesting model sees.

Docs

Quickstart · Plug in your agent · Action verification · Retrospective analysis · Calibrating the judge · The safety pack · Writing personas · Running in CI · Claude Code plugin · Evaluation intelligence

License

MIT. pip install claque