Designing Vibe Tests

A playbook for an AI agent designing a vibe test for a user. The hard part isn't running the test — it's turning a vague "I want to test X" into a runnable, fair evaluation. That happens through a short interview. This page teaches the agent what questions to ask.

A vibe test is a structured, fair evaluation that measures how well an LLM performs a task under different configurations: same prompts, different conditions, measurable outcomes. It's a compass for a fast-moving space — prototype the options, throw naive prompts at them, and let data (not opinion or memory) decide.

This is the general design methodology and applies to any decision — a tool's output format, a skill's wording, a workflow, a model — not just component APIs. For the two Astryx-specific applications, see API Arbitration and Vibe Evaluation.

The Agent's Job: Run the Intake

When a user says "I want to vibe test X," do not start writing prompts. First run a short interview that pins down four ingredients. For each one, propose a sensible default and ask the user to react — reacting to a concrete proposal is far easier for a user than authoring a spec from a blank page.

The four ingredients:

#	Ingredient	The question you're really answering
1	Goal & measure	What decides the winner?
2	End-user inputs	What does a naive prompt look like?
3	Variations	What are we actually comparing?
4	Orchestration & judging	Who runs it, and who scores it?

Work through them in order — each answer feeds the next.

Ask early: ad-hoc or stored?

Not every vibe test needs a pipeline. Pin this down during the intake, because it changes how much scaffolding you build:

	Ad-hoc (most tests)	Stored / scheduled
Purpose	Settle a question now — an API shape, a doc wording, "does this help?"	Track a system over time; a nightly benchmark
Results	Just show a results table in chat / the issue / PR	Structured JSON per run, aggregated + reported
Overhead	None — no schema, no storage, no reporting job	Result schema, storage path, report/self-healing loop
Reuse	Throwaway once the decision is made	Compared across runs; trend matters

Default to ad-hoc. Only build the #Presenting Results JSON schema and #Reporting and Self-Healing (stored tests only) machinery when the user actually needs to compare across runs or run it on a schedule. For a one-off decision, a table is the deliverable.

1 — Goal & measure: "What decides the winner?"

Ask:

What does "better" actually mean here — the downstream outcome for the user, or the artifact looking nice? These often diverge.
Is this a quality question, a cost (token/latency) question, or both?
What failure mode are you worried about? (That failure usually becomes a scored dimension.)

Then propose the dimensions and the tie-breaker before running anything. Pick dimensions that capture the real tradeoffs:

Domain	Example dimensions
UI / code generation	Correctness, accessibility, code quality, efficiency, maintainability
Tool / CLI usage	Accuracy, speed, self-healing, token efficiency
Skill / doc quality	Discoverability, hallucination rate, verbosity, unused features
Any domain	Success rate, duration, token cost, error recovery

2 — End-user inputs: "What does a naive prompt look like?"

Ask:

What do users actually ask in this situation? (The real downstream task — not "use tool X.")
What's the range from simple to synthesizing/multi-step?
What are the nasty inputs — missing data, ambiguity, edge cases, coverage gaps?

Then propose a prompt battery of 5–15 prompts that describe the user experience and never name the solution:

❌ Names the solution	✅ Describes the experience
"Create an accordion with CollapsibleGroup"	"Build a FAQ page where users expand each question"
"Return the receipt as markdown"	"Split this bill across 3 people including tip"

Cover easy wins (sanity), differentiators (where one arm pulls ahead), edge cases, and coverage gaps (things an arm can't do at all — real signal). Use stratified sampling so categories stay balanced.

3 — Variations: "What are we actually comparing?"

Ask:

Just the options you named, or is there an obvious third? A hybrid arm (both/all options available) reveals which one the LLM naturally reaches for.
What is the single variable? Everything else must be held identical across arms (same tool description, same doc length/structure) so the comparison is fair.

Then propose 2–5 conditions, each constrained to only what it's being tested on. Constraints are what make the comparison meaningful.

4 — Orchestration & judging: "Who runs it, who scores it?"

Ask:

Isolated sub-agents (higher confidence, more expensive) or a single-model quick pass (directional signal only, acknowledged contamination)?
Is the outcome objectively checkable (script it against ground truth) or fuzzy (needs a judge agent)? Often it's both.

Then propose the run shape: one isolated sub-agent per (prompt × condition) pair, a deterministic checker for anything scriptable, and a separate judge agent for the fuzzy dimensions. Sub-agents never score their own work.

Worked Example: Starting the Intake

A concrete run of the interview. Notice the agent proposes a default at every step so the user just reacts.

User: "We're thinking about shipping a parse_receipt tool for our agents. Should it return raw JSON or a pre-formatted markdown summary? Markdown feels nicer but I'm not sure."

1 · Goal & measure —

Ask: Do we judge the agent's final answer to the user, or the tool output itself? Any failure you're worried about? Propose: "We measure the agent's final answer to the end user, not the tool output. Dimensions: correctness (right total/date/vendor?), hallucination rate (invented a field?), and token cost. Correctness breaks ties."

2 · End-user inputs —

Ask: What do people actually ask after a receipt is parsed? Any nasty inputs? Propose: "~10 downstream prompts, none mentioning JSON or markdown: 3 simple lookups, 4 requiring arithmetic over line items, 3 edge cases (missing field, foreign currency, two receipts in one image). Same receipt fixtures across all arms."

3 · Variations —

Ask: Just the two, or add a hybrid? Should only the return payload differ? Propose: "Three conditions — json-only, markdown-only, hybrid (both). Identical tool descriptions; the return value is the only variable."

4 · Orchestration & judging —

Ask: Isolated sub-agents or a quick single-model pass? Correctness is checkable against fixtures — script it? Propose: "30 isolated sub-agents (10 prompts × 3 conditions). A scripted checker grades correctness against ground-truth fixtures; a separate judge agent scores hallucination and reads traces for self-healing. No self-scoring."

Once the user reacts to those four defaults, the test is fully specified:

QUESTION:   Does parse_receipt's return format change answer quality?
            (json vs markdown vs hybrid)

DIMENSIONS: correctness (scripted, ground-truth) · hallucination (judge)
            · token cost (measured)          — correctness wins ties

PROMPTS:    10 downstream tasks, no format words
            3 simple · 4 arithmetic · 3 edge     (same fixtures, all arms)

CONDITIONS: json-only | markdown-only | hybrid   (payload = only variable)

RUN:        30 isolated sub-agents (prompt × condition)
JUDGE:      scripted correctness + 1 judge agent for hallucination/traces
STORE:      results JSON per run → aggregate → report

Sample of the battery — note the zero format language:

#	tier	prompt
p01	simple	"What did I spend at this store?"
p04	arithmetic	"Split this bill evenly across 3 people, including the tip."
p08	edge	"Which of these two receipts was more expensive?"

Non-Negotiables (get these wrong and the test is invalid)

Even a well-designed test dies on these. Hold the line during the intake.

Same prompts across every condition

Every arm runs the exact same prompts. Sample once, reuse the identical set everywhere. Otherwise you're comparing apples to oranges and the results are noise.

Constrain each condition fairly

Each agent gets only what it's being tested on. The single variable is the thing under test; everything else is held identical. A hybrid arm (all options available) is the clean way to see which the LLM reaches for.

Simulate a naive environment

Testing agents must behave like a user with no prior knowledge:

Prompts contain no leading language and never name the solution.
Sub-agents inherit no context — no SOUL.md, MEMORY.md, USER.md, or conversation history that could leak the answer. See #Sub-Agent Isolation.
Prompts cover common and edge cases so you don't declare victory on the happy path.

Judge separately and comparatively

A dedicated judge that sees all arms side-by-side scores far more reliably than sub-agents grading themselves. Self-judging kills the ability to spot differences.

Store what you'll reuse

For stored/scheduled tests, persist prompt batteries, per-run JSON, and reports — future runs need to compare and reproduce, and the trend matters more than any single run. For ad-hoc tests, skip this: a results table is the whole deliverable. Do keep the prompt battery if you think you'll want to re-run the same comparison later.

Sub-Agent Isolation

The most common way to get invalid results. A sub-agent that inherits the parent's context isn't testing "can an LLM discover the right pattern?" — it's testing "can an LLM that already knows the answer confirm it?"

Signs of contamination:

Zero hallucinations across all arms (real naive agents hallucinate sometimes).
All arms score nearly identically (if the agent already knows, wording doesn't matter).
Uniform high confidence with no hedging (real discovery involves uncertainty).
A single agent processed multiple arms sequentially.

How to ensure isolation:

Spawn each agent directly from the parent — don't nest.
Make the task fully self-contained (reference path, prompt, output path, schema).
Constrain explicitly (see template).
One agent per (prompt × condition). Never batch.

Isolation prompt template

You are running a vibe test.

You have NO prior knowledge of the system under test. Do NOT use prior
knowledge of any specific library, tool, product, or convention.

## Reference
Read the reference at: {path}
This is your ONLY reference. Use ONLY what is documented there.

## Task
{prompt}

## Output
Write your result to: {result_path}
Report: what you reached for first, where you hesitated, anything you
wanted but couldn't find, and any workarounds you used.

Agent as Judge

Split evaluation into two roles.

Sub-agents — generate + self-report (no scores): the solution, the approach and why, what they relied on, what they were uncertain about, any workarounds.

Judge agent — comparative scoring + analysis: receives all arms' outputs for each prompt side-by-side, plus the original prompt and rubric. Scores every arm on the same scale and explains which was better and why. It works because the judge has cross-condition visibility and calibrated scoring (a 4 and a 5 are relative to each other).

Judge prompt template

You are evaluating outputs from a vibe test comparing {N} conditions.

For each prompt you receive: the original prompt, the output from each
condition, and each agent's self-reported notes.

Score EACH condition on: {your dimensions, each 1-5}.

For each prompt, write: scores per condition, a qualitative comparison
(what each did differently and why), and which was better + why.

After all prompts: an overall recommendation with evidence.
Output JSON to: {result_path}

Judge vs self-eval

Scenario	Method
Single-condition evaluation	Self-eval is fine
Any cross-condition comparison	Judge required
Trend tracking (same condition over time)	Self-eval acceptable
Objectively checkable outcome	Script it; judge only the fuzzy parts

Presenting Results

Ad-hoc → just show a table

For a one-off decision, the deliverable is a results table — no schema, no storage, no reporting job. Show the judge's scores per condition and a one-line verdict. That's enough to make the call and cite it in a PR or issue comment.

| Prompt        | json-only | markdown-only | hybrid | Notes                          |
|---------------|-----------|---------------|--------|--------------------------------|
| p01 lookup    | 5         | 5             | 5      | all correct                    |
| p04 arithmetic| 3         | 5             | 5      | json arm dropped a line item   |
| p08 edge      | 5         | 2             | 5      | markdown lost the 2nd receipt  |
| **avg**       | **4.3**   | **4.0**       | **5.0**| hybrid: markdown UX + json safety |

Verdict: ship hybrid — markdown for the summary, raw JSON so nothing is lost.

Keep the prompt battery around if you think you'll re-run the comparison later; otherwise it's throwaway.

Stored/scheduled → structured JSON

Only when you need to compare across runs or run on a schedule. Each sub-agent writes structured JSON; adapt fields to your domain (the tool-call trace and self-report are always worth keeping).

{
  "promptId": "p01",
  "condition": "condition-name",
  "prompt": "the full prompt text",
  "durationMs": 12345,
  "output": "the result or generated code",
  "success": true,
  "toolCalls": [
    { "index": 1, "command": "...", "outputPreview": "...", "error": null, "isRetry": false }
  ],
  "inputTokens": 5000,
  "outputTokens": 2000,
  "hallucinated": false,
  "errorRecoveries": 0,
  "notes": "what it reached for, where it hesitated, workarounds used"
}

Reporting and Self-Healing (stored tests only)

Skip this section for ad-hoc tests — a results table is enough.

For stored/scheduled tests, results should live somewhere referenceable — an issue, a scored wiki page, a report — so they can be cited in PRs and future decisions. Consider running vibe tests on a schedule (a nightly "night watch") so regressions surface automatically.

One trap when acting on failures: naively fixing the single issue an agent hit tends to over-fit to that one case. Require recurring patterns across multiple runs, and reproduce the issue before shipping a fix.

Key Lessons

Build it, then test it. Don't debate options in the abstract — prototype, then feel the friction with data.
Naming matters hugely for LLMs. The same functionality scores very differently under different names — a sensitivity humans don't share.
Measure the cost side too. Brief docs/outputs can match richer ones at a fraction of the tokens.
Coverage gaps are real signal. An arm that can't do something at all is worth knowing — don't skip boundary prompts.
The hybrid arm reveals the natural choice. When a both-options agent consistently picks one, that's your answer.
Same prompts, always. Without identical prompts across arms, results are noise.
Self-healing is a dimension. Graceful failure and recovery often beats acing easy prompts and crashing on hard ones.
Watch for contamination. Zero hallucinations and uniform confidence usually mean the agent already knew the answer — not that your system is perfect.

Uh oh!

Designing Vibe Tests

Designing Vibe Tests

The Agent's Job: Run the Intake

Ask early: ad-hoc or stored?

1 — Goal & measure: "What decides the winner?"

2 — End-user inputs: "What does a naive prompt look like?"

3 — Variations: "What are we actually comparing?"

4 — Orchestration & judging: "Who runs it, who scores it?"

Worked Example: Starting the Intake

Non-Negotiables (get these wrong and the test is invalid)

Same prompts across every condition

Constrain each condition fairly

Simulate a naive environment

Judge separately and comparatively

Store what you'll reuse

Sub-Agent Isolation

Isolation prompt template

Agent as Judge

Judge prompt template

Judge vs self-eval

Presenting Results

Ad-hoc → just show a table

Stored/scheduled → structured JSON

Reporting and Self-Healing (stored tests only)

Key Lessons

Checklist: Before Running a Vibe Test

Related

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!