Skip to content

Design Principles and Guarantees

Jump to bottom
Arnaud Manaranche edited this page Jul 8, 2026 · 1 revision

Design Principles & Guarantees

Why AI Feature Pipeline (AFP) is more than "an agent that writes code" — and what it actually guarantees.

Most AI coding agents take the shortest path from prompt to diff: no scoping, no design review, no QA, no memory of what was tried before. That's fine for a throwaway script and dangerous for a real codebase.

AFP forces the same discipline a human team applies before merging. But discipline alone isn't enough to trust a system with your repo. Three things separate a demo from something a team can lean on:

Pillar The question it answers
Trust Do the guardrails actually hold — or are they just suggestions to the model?
Cost & sustainability Does this stay viable after 200 features, or does the repo rot?
Measurability Do I know whether a change improves the pipeline, or just changes it?

This page explains how AFP answers each.

Core stance: control flow is deterministic, not model-decided

The single most important design choice: gates are code, not prompts. Retry, halt, handoff, and budget limits are deterministic checks in run-pipeline.sh — never something the agent decides for itself. A model that phrases its output differently still produces a schema-valid result; a model that "feels" a plan is fine still can't skip the design gate.

This is the opposite of autonomous multi-agent frameworks that let the model drive orchestration — which is exactly where they drift. AFP treats every piece of model output as untrusted input and validates it in code.

Pillar 1 — Trust: guarantees enforced in code, not prose

A rule that only lives in a prompt is a suggestion. AFP moves the critical ones into code:

  • Structured output, validated. Every agent's response is a forced submit_changes tool call validated against a per-role JSON Schema — and the required fields are re-checked server-side, because forcing which function is called doesn't force its arguments to be valid. No silent "0 files parsed" from formatting drift.
  • Per-role write permissions, enforced independently of the prompt. PM/Architect/Review/QA can't write source files at all; only Dev can, and only to configured extensions. Checked against every path in the model's own output before anything touches disk.
  • Path containment. Every model-supplied path is resolved and prefix-checked against the project root — a ../ traversal is refused even if the extension looks legitimate.
  • Design gate bound to content. No code is written until a human approves the technical plan. The approval is bound to the plan's content hash: a resumed run reuses the exact plan the human reviewed (it does not regenerate a non-deterministic one), and a plan that changes afterward forces re-approval.
  • No placeholders / no committed secrets — as gates. These were prompt-only rules a truncating model could ignore. Now deterministic scans run over Dev's own changed files (placeholder/truncation markers; private keys, cloud/provider tokens, assigned credential literals) and feed the same retry loop.
  • Hook-respecting commits. The pipeline never uses --no-verify; a real pre-commit rejection stops the run instead of being bypassed.
  • Optional adversarial review. A single reviewer is the weakest gate — nothing contradicts a plausible-but-wrong PASS. Set review.verifiers > 1 to run N independent passes with distinct lenses (correctness/AC → diff-vs-diagram → edge-cases/security) and decide by majority.

What this lets you say: these rules are enforced — not "the prompt asks the model to."

Pillar 2 — Auditability: every output is traceable

Each stage's commit carries provenance trailers:

AFP-Model: anthropic/claude-...
AFP-Prompt-SHA: 9f2ac1b0e4d3

For any artifact or diff, git log answers "which model and which prompt version produced this?" A prompt edit surfaces as a changed hash on subsequent runs. Combined with the per-feature retrospectives, this turns the pipeline's history into an audit trail — the argument that matters for a code review, a lead, or a compliance check.

Pillar 3 — Sustainability: bounded growth

The pipeline writes two kinds of file, and only one belongs in git long-term:

  • Durable knowledge (kept): feature briefs, technical plans, reviews, QA reports, retrospectives, project-memory.md, the memory-compact counter, the design-approval hash.
  • Derived / per-run debug (ignored): context.json (rebuilt from source every run) and the per-feature .agent-* files — raw submit_changes payloads (full file contents, often hundreds of KB), status flags, manifests, token accounting.

afp-setup installs a .ai/.gitignore so a project shipping hundreds of features doesn't accumulate LLM dumps and a regenerable index in its history. project-memory.md itself stays bounded by design — four fixed categories plus periodic memory-compact deduplication, not one section per feature forever. A maintenance script (prune-artifacts.sh) untracks legacy debug files and archives shipped feature folders.

The point: this is debt that no longer accumulates.

Pillar 4 — Measurable self-improvement (human-gated)

Discipline without measurement is just drift. AFP closes a measurable loop for improving its own prompts — without an autonomous agent silently rewriting itself.

  1. Vary a prompt without editing agents.json: AFP_SKILL_<ROLE>=.ai/experiments/pm-v2.md. The recorded prompt hash reflects the variant, so the experiment is auditable.
  2. A/B the output against a baseline, scored by the same rubric (the eval harness). A candidate "regresses" if its score drops or any previously-passing check now fails — so a change that fixes one thing while quietly breaking another is still caught.
  3. Track score over time per prompt version with an append-only history log.

You keep the winning prompt or discard it — the decision stays human, but it's now backed by a number and traceable to an exact hash.

The two memories

AFP deliberately separates two kinds of memory so agents don't rediscover the project on every run:

  • Derived memory — .ai/context.json: a symbol/dependency index rebuilt from source (incrementally, fingerprinted by mtime). Always correct, never hand-maintained, never versioned.
  • Earned memory — .ai/project-memory.md: pitfalls, confirmed conventions, architecture decisions, integration notes, written by the Retro and read by every role. This is the knowledge no static analysis can give you — and the one that must be curated and bounded.

How a feature flows

 SCOPE     CLARIFY      DESIGN       BUILD      REVIEW      VERIFY      LEARN
  PM   →  Dev Review → Architect →   Dev    →  Review   →   QA     →   Retro
 brief     Q&A loop     plan+diagram  code     verdict     verify     memory
                         │                       │
                    design gate            adversarial
                   (human approves)         panel (opt.)

Every run executes in an isolated git worktree — fully reversible, never touching your working checkout — behind a concurrency lock and a per-feature token budget circuit breaker.

Honest limits

  • It doesn't make agents smarter — it makes their mistakes harder to let through.
  • The eval is structural by default (sections present, diagram present, verdict recorded); judging business correctness needs the opt-in LLM-judge, which costs money and a key.
  • The improvement loop is measured, not autonomous — a human still promotes a winning prompt. That's a governance choice, not a technical limit.
  • A human still reviews the design and merges. AFP never merges or closes anything.

In one sentence

AFP turns "an agent that writes code" into a system that is verifiable (the rules hold), auditable (you know what produced what), sustainable (the repo doesn't rot), and improvable by measurement (you know if a change is progress) — which is exactly what it takes to dare point an AI agent at real code.

Clone this wiki locally