Skip to content

SLM Harness

Jump to bottom
Your GitHub Name edited this page Jun 17, 2026 · 7 revisions

SLM Harness

PromptPilot uses a small language model as the harness around a frontier coding agent.

Why use an SLM?

Many coding-agent sessions contain control decisions that do not require a frontier model:

  • Is this prompt ambiguous?
  • Is this mostly repeated log output?
  • What are the important file paths?
  • What user constraints must be preserved?
  • Should the prompt be passed through unchanged?
  • Is this a simple answer that does not require agent execution?

The SLM controls the workflow; it does not replace the coding agent.

What the SLM is trusted to do

  • Extract constraints
  • Classify intent
  • Detect ambiguity
  • Compress repetitive output
  • Recommend route
  • Preserve structured facts
  • Flag high-risk transformations for passthrough

What the SLM is not trusted to do

  • Implement complex code changes
  • Debug deep logic bugs
  • Infer hidden product requirements
  • Drop constraints to save tokens
  • Make irreversible changes
  • Override explicit developer instructions

Fallback principle

When uncertain, passthrough.

The SLM is useful only when it reduces noise without changing intent. A slightly more expensive run is better than a cheap but wrong run.

Harness outputs

A successful harness result should make the downstream agent's job clearer while preserving meaning. The SLM emits an ExecutionSpec whose route is one of four values:

  • clarify — a clarification question when the request is ambiguous.
  • answer — a direct SLM reply when no coding-agent execution is needed (only surfaced interactively when the user opts in via --let-slm-answer).
  • passthrough — pass the raw prompt unchanged when rewrite risk is high.
  • act — invoke the coding agent with a constraint-preserving downstream_prompt.

A separate PostToolUse hook (not part of the SLM's route choice) compresses noisy bash tool output — pytest traces, grep floods, installer logs, large diffs — using regex/heuristic rules that keep failures, paths, stack frames, commands, and API boundaries. See Tool Output Compression.

Output format: prose envelope vs JSON spec

PromptPilot has two SLM output formats. The downstream coding agent always receives plain text (the rewritten prompt) — the format below is what the SLM emits to PromptPilot, which then extracts the text and forwards it.

Legacy: prose envelope (v1)

--normalizer slm-anthropic, --normalizer slm-openai, and --normalizer slm-subscription use this format. (The auto-detected default slm now prefers the v2 JSON spec below for every backend — see "Default".) The SLM emits a structured-header preamble, a --- separator, and the rewritten prompt:

INTENT: act
SCOPE: localized
---
Fix the failing auth test without changing public API behavior or migration files.

PromptPilot parses the header lines, then forwards everything after the separator to the coding agent. Simple, regex-based parser; no SDK dependencies.

Default: JSON ExecutionSpec (v2)

--normalizer slm-anthropic-v2, --normalizer slm-openai-v2, and --normalizer slm-subscription-v2 make the SLM emit a single JSON object carrying the full execution spec (prpt/core/spec.py#L27). The auto-detected slm backend selects the v2 normalizer matching your auth (API key or Max OAuth / ChatGPT subscription), so this is the format most runs use — and the only one that can route clarify or passthrough:

{
  "route":            "act",
  "intent":           "act",
  "scope":            "localized",
  "needs_history":    false,
  "context_policy":   "targeted",
  "target_files":     ["src/auth/login.py", "tests/test_auth.py"],
  "risk":             "low",
  "downstream_prompt": "Fix the failing auth test without changing public API behavior or migration files.",
  "memory_record":    "User wants the failing auth test fixed; preserve public API and migrations."
}

PromptPilot extracts downstream_prompt and forwards it to the coding agent exactly as in v1; the other fields drive internal routing (route → clarify / answer / passthrough / act), context loading (context_policy, target_files), and session memory (memory_record becomes the next turn's assistant record).

The JSON is the SLM→PromptPilot wire format only — the coding agent receives plain text, never the spec. To see the spec itself, run prpt --show-spec "..." (prints the parsed ExecutionSpec to stderr) or set PROMPTPILOT_V2_RAW_LOG=1 to log each raw model response to ~/.promptpilot/v2_slm_raw.jsonl.

Why two formats?

The prose envelope carries only two structured fields (INTENT + SCOPE). The JSON spec carries nine. v2 was added when the harness grew features that needed more than INTENT/SCOPE to drive — the routing decision (route), constraint-preservation hints (target_files, context_policy), risk classification (risk), and a session-memory record (memory_record). JSON also parses more reliably than free-form prose at the cost of a slightly heavier system prompt.

Fail-open by design. If the v2 normalizer's JSON parse fails, the code falls back to the v1 prose parser (prpt/normalizers/slm_openai_v2.py#L144-L165). If both fail, heuristic defaults (intent=act, scope=localized) keep the pipeline running. The downstream coding agent never sees broken JSON.

v2 is the default across every auto-detected backend — slm-anthropic-v2 / slm-openai-v2 for SDK keys, slm-subscription-v2 for Max OAuth / ChatGPT — because the routing decision (route), constraint hints (target_files, context_policy), risk, and memory_record it carries drive most of the harness, and clarify/passthrough only exist on the v2 path. v1 stays as the JSON-parse fallback and an explicit opt-out (--normalizer slm-anthropic / slm-openai / slm-subscription) for the simpler prose envelope. The mapping from route values to harness behavior is documented in Routes and Decisions.

Which SLM model?

The SLM's output (the rewritten downstream_prompt and the memory_record) is re-fed to the coding agent every turn, so in principle SLM terseness sets how lean the agent's input is.

Whether a bulkier SLM actually inflates the agent's input is UNVERIFIED. The numbers below were collected in separate runs with different cache warmth (the nano run ran ~89–94% cache-hit, the mini run ~78–90%) and different normalizers/transport, so the uncached figures are not comparable (cross-run/cache-confounded — see MEASUREMENT_METHODOLOGY.md). On cache-independent total tokens the mini runs actually fed slightly fewer tokens, so the data does not support "mini is bulkier / nano is terser."

SLM model with_session slm_native
gpt-5.4-nano 118,610 190,578
gpt-5.4-mini 185,677 326,671

⚠️ Cross-run, cache-confounded — not a clean result. The two rows come from different runs with different cache warmth, so the apparent gap is an artifact of caching, not SLM terseness. A clean interleaved same-run comparison has not yet been done; the nano-vs-mini token-efficiency question is open.

End-state was parity — every config fixed the bug — so model choice did not affect quality in these runs.

Transport is agent-uncached-invariant

How the SLM is called — an API SDK call vs. a subscription CLI subprocess — does not change the agent's input. For the same SLM model the agent sees the same tokens regardless of transport: claude Haiku via API was 67,501 uncached vs. 65,608 via Max subscription (~3%, equal). Transport only changes where the SLM cost lands ($ on an API key vs. quota on a flat subscription) and adds ~20k tokens/call of CLI overhead on the subprocess path. So choose the SLM model for agent cost, and the transport for billing — they are independent knobs.

The hybrid setup is preferable to going all-subscription, on two valid grounds: running the SLM on a cheap API key avoids the ~20k tokens/call of CLI subprocess overhead that the all-subscription path incurs (see Transport above), and it puts the SLM cost on predictable metered dollars instead of finite subscription quota. The default SLM differs per backend — OpenAI API → gpt-5.4-nano; codex/ChatGPT subscription → gpt-5.4-mini; Anthropic API and Max → claude-haiku-4-5.

Example

This is a schematic example showing both formats for the same input.

Raw prompt:

Fix the failing auth test. Keep the public API stable and do not touch migrations.

Legacy (v1 prose envelope — explicit opt-in):

INTENT: act
SCOPE: localized
---
Fix the failing auth test without changing public API behavior or migration files.

Default (v2 JSON spec):

{"route":"act","intent":"act","scope":"localized","needs_history":false,
 "context_policy":"targeted","target_files":["src/auth/login.py"],"risk":"low",
 "downstream_prompt":"Fix the failing auth test without changing public API behavior or migration files.",
 "memory_record":"User wants failing auth test fixed; preserve public API + migrations."}

Either way, the coding agent receives only the downstream_prompt text. The SLM clarifies the work envelope; the coding agent performs the implementation.

See also: Routes and Decisions · Semantic Preservation · Safety Model

Clone this wiki locally