Simple Experiment Loop

A zero-admin skill for giving coding agents a tight, measurable inner loop:

establish red -> mutate -> measure -> repeat until green

It is deliberately small. There are no manifests, viewers, orchestration layers, or persistent state. The agent works against one local terminal command and stops when that command succeeds or the iteration cap is reached.

Why use it?

Modern coding agents often find the right change immediately. The value of the loop is not forcing extra attempts; it is requiring evidence:

• reproduce the failure before editing;
• freeze the files the agent may change;
• protect the feedback gate from reward-hacking;
• measure every mutation; and
• report the baseline-to-final delta.

Use Search Mode when the solution is uncertain and the agent needs to hill-climb through failures. Use Verification Mode when the likely fix is obvious but you still want proof that it works.

Featured example: 1,000x faster

The performance challenge starts with a correct but quadratic duplicate matcher:

Target: matcher.py
Feedback Command: python -m pytest -q test_matcher.py

RED   5000 matches in 878 ms (budget: 50 ms)
MUTATE replace nested scans with set membership
GREEN 5000 matches in 0.803 ms

The agent solved it in one mutation. That is a successful loop: the regression was reproduced, the 50 ms threshold came from the test contract, the change stayed inside the Target, and the improvement was measured rather than asserted.

Try it in examples/02-performance, or see all four challenges.

Install

Clone the repository, then run the installer for your platform:

git clone https://github.com/cauldnz/simple-experiment-loop.git
cd simple-experiment-loop
./scripts/install.sh

git clone https://github.com/cauldnz/simple-experiment-loop.git
Set-Location simple-experiment-loop
.\scripts\install.ps1

The installer writes the skill to both:

• ~/.claude/skills/simple-experiment-loop/ with canonical YAML frontmatter; and
• ~/.copilot/skills/simple-experiment-loop/ with the XML header expected by GitHub Copilot CLI.

Existing installations are left untouched unless you explicitly pass --force on Bash or -Force on PowerShell.

Use

Give your agent the Target and Feedback Command when they are known:

Use the simple-experiment-loop skill.
Target: matcher.py
Feedback Command: python -m pytest -q test_matcher.py

If no feedback command exists, the skill allows the agent to author the gate before taking the red baseline. Once the loop starts, the gate and Target are frozen.

The skill works best when feedback is deterministic and local: tests, compilers, linters, benchmarks, or bundle-size checks. For subjective work, first turn the desired qualities into fixed cases and actionable, objective faults.

Example fleet

Challenge	Feedback Command	Demonstrates
Messy CSV parser	python -m pytest -q test_parser.py	Edge cases expose a naive parser
Performance	python -m pytest -q test_matcher.py	A hard budget drives an algorithmic change
Strict TypeScript	npm test	Compiler feedback reveals faults progressively
Async retries	npm test	Named checks enforce backoff and retry limits

Each committed example begins red so another human or agent can run the loop themselves.

Design boundary

Simplicity is the product. Changes should remove a demonstrated ambiguity or failure without adding ceremony. The maintainer invariants are documented in AGENTS.md.