Eval-driven prompt optimization for small local models (measured on the #69 harness)

[hellices]

Why

PR #172 fixed the worst observed small-model failure modes (composite namespace/name 404 loops, idle screen), but the prompts have still never been tuned against measured results — the #69 harness landed after the prompt wording did, and no eval-driven prompt iteration has happened since. The wording in agent/prompts.py is best-effort, not evidence-backed.

Industry check (verified 2026-08, across aider, Cline, OpenHands, smolagents, Goose, RA.Aid, Continue.dev, opencode): no product maintains separate per-model system prompts. The dominant pattern is one shared prompt template plus per-model settings (tool-call format, temperature, capability flags) — aider's model-settings.yml is the clearest expression. korvid's full/small capability-profile design (#71) is already the right architecture; what's missing is closing the loop between eval failures and prompt wording.

What

An eval-driven prompt optimization pass, measured before/after on the #69 harness with the bundled 24-scenario pack (≥3 reps), against at least two small models (e.g. qwen3:4b, qwen3:8b — current BFCL open-weight FC leaders at these sizes).

1. Baseline: run python -m korvid.evals --profile small (and full on the larger model) per model; commit the reports as the "before" numbers.
2. Failure-mode analysis: classify failed runs from the event streams — wrong tool, wrong arguments, gave up early, answered without evidence, misdiagnosis keyword, iteration exhaustion.
3. Targeted prompt edits in agent/prompts.py (SMALL_SYSTEM_PROMPT, SMALL_TOOL_DESCRIPTIONS, and the shared clauses where evidence points there), one failure mode at a time. Candidate levers, each only if the eval shows the failure:
   • worked-example coverage (the single OOM example may over-anchor answers toward OOM),
   • explicit stop-condition wording ("answer once the evidence names the cause; do not keep exploring"),
   • negative-control handling ("say the resource is healthy when checks pass" — the pack has 3 healthy scenarios),
   • diagnose_pod-first routing strength.
4. After: re-run the same matrix; a change lands only with a non-regressing overall success rate and an improved target metric. Numbers go into the PR description and the scoreboard (companion issue).

Invariants

• Absolute invariants from Agent eval harness: scenario-based diagnostic benchmarks for small local models #69 hold in every configuration: safety violations = 0, malformed calls < 1%.
• Frontier/full-profile behavior must not regress: the full-profile prompt only changes if its own numbers improve.
• Prompt-invariant tests in tests/agent/ pin any new behavioral wording.

Out of scope

• Per-model prompt files (anti-pattern per the industry check; capability tiers are the unit of prompt variation).
• New profile tiers or automatic capability detection (revisit only if the eval shows the two-tier split is insufficient).
• Tool-format fallbacks for models without native tool calling (e.g. Gemma 3) — separate concern, needs its own design.

[hellices]

Session results: six measured prompt iterations (local Ollama, M4 16 GB)

Ran the full loop this issue asks for, against live models. All runs: bundled 23-scenario pack, --profile small, deterministic grading.

Headline numbers

configuration	success	evidence	malformed	safety
qwen3:4b, baseline prompt (full pack, 1 rep)	17/23 (74%)	18/23	0/31	0
qwen3:4b, optimized prompt (full pack, 1 rep)	18/23 (78%)	18/23	0/32	0
qwen3:4b, targeted subset ×3 reps: baseline → final	4/10 (40%) → 22/30 (73%)	—	0	0
llama3.2:3b, baseline (full pack, 1 rep)	5/23 (22%)	9/23	1/22	0
llama3.2:3b, optimized (full pack, 3 reps)	17/69 (25%)	24/69	2/70	0

Single-rep full-pack numbers are noisy (observed 16–20/23 for identical configs); the trustworthy signal is the targeted subset (the 8 measured failure scenarios + 2 regression guards, oom-killed and crashloop-app-panic) at 3 reps: 40% → 73%, with liveness-probe-failing, oom-killed, quota-blocked-scheduling, job-backoff-limit-exceeded and crashloop-app-panic all at 3/3.

What was measured, and what fixed it

1. Worked-example anchoring is the strongest lever. The example's exit=137 (OOMKilled) caused liveness kills to be misdiagnosed as OOM — including verbatim parroting of the example answer. Rules alone ("the reason string, not the exit code") did not fix it; a "method-only, never reuse" marker did not fix it. Rewriting the example to demonstrate the 137 discrimination itself (event reason decides) fixed it at 3/3.
2. Whack-a-mole between anchors is real. The liveness example dropped the exit-code-citation habit and oom-killed fell 3/3 → 1/3 (answers stopped naming "137"). Fix: the example answer cites the quoted event reason and last exit 137 together.
3. Contrastive rule-outs trip the misdiagnosis gate. "Never claim OOM" made the model write "not OOMKilled, but…" — which still names OOM outside any negation scope. The working rule is one root cause, never name faults you ruled out.
4. Healthy verdicts need mechanical wording. "Say it's healthy" lost to hedging; "start your answer with healthy and name the checks that pass; a ready pod is healthy even after restarts" recovered the negative controls (healthy-restart-history remains the stubborn one — the 255-exit history keeps luring a fault verdict).
5. Prompt changes don't rescue capability floors. llama3.2:3b moved 22% → 25% (noise): its failures are evidence-fetch failures (9/23 baseline), not wording. Scoreboard-relevant: below ~4B with weak FC, the small profile can't compensate.

Commits: c9fac2a (round 1), 1ef0bb1 (rounds 2–4), 81cbb67 (rounds 5–6). Every rule is pinned in tests/agent/test_profiles.py::test_small_profile_prompt_pins_measured_failure_mode_rules with the measured failure it answers. Invariants held throughout: safety violations 0 everywhere, malformed calls ≤ 1% for qwen3:4b.

Remaining measured failures (next round's targets)

• pvc-pending-no-storageclass 0/3 — the model stops at "unbound PVC" and never fetches the PVC to see the missing storage class (the pointer-chasing rule is too weak for a two-hop chain).
• healthy-restart-history 1/3 — stopped-restart history still reads as a live fault.
• stuck-rollout ~1–2/3 — the ReplicaSet→pod hop to find ImagePullBackOff is inconsistently taken.
• Raw reports (baseline/rounds/final JSON+MD) are retained as session artifacts; publishing them in-repo is Local/on-prem agent quality: conversational AKS exploration eval, model scoreboard, prompt/tool tuning, and AKS matrix #176.

[hellices]

Next round: deterministic evidence tools + AKS model matrix

Root cause and implementation

The remaining failures were not solved by more prompt text:

• diagnose_pod fetched PVC phase only, omitting storageClassName and the PVC's own ProvisioningFailed event.
• recovered pods showed previous-instance error logs without an explicit current-health verdict.
• Deployment inspection stopped at ProgressDeadlineExceeded; the model had to discover Deployment -> ReplicaSet -> Pod over multiple calls.

Implemented in commits 65bbbcc, e548e73, 88116d4, b986103, and review fix 16febdd:

• diagnose_pod: CURRENT HEALTH projection (HEALTHY NOW, UNHEALTHY NOW, COMPLETED SUCCESSFULLY), PVC phase + three-state StorageClass (named / defaultable / explicit none) + PVC warning events.
• new read-only diagnose_workload: Deployment conditions/events, immutable-UID owner traversal to ReplicaSets and non-ready pods, bounded embedded pod diagnoses (max 3; <=8k total); one disappearing pod no longer discards parent/sibling evidence.
• full/small/MCP exposure, concise small-profile description, MCP follow mapping.
• eval fixtures repaired with real ownerReference.uid values and integrity checks; compound routes are accepted as alternative evidence paths.

AKS discovery and execution

Found the existing deployment in subscription ME-MngEnvMCAP310512-inhwanhwang-3:

• cluster aks-shared-runners, namespace/deployment/service ollama/ollama
• persistent 30Gi PVC ollama-models
• original node: Standard_D2s_v5 (Ollama limit 3.5 CPU / 10Gi)
• public LB exists but timed out externally; eval used kubectl port-forward

GPU finding: Korea Central exposes T4 SKUs but Standard NCASv3_T4 Family quota is 0; an automated request for 4 vCPU returned QuotaNotAvailableForResource. Created a temporary zone-2 Standard_D16s_v5 Spot pool instead (matching the zonal model PVC), ran evals, restored the original deployment settings, then deleted the pool. No temporary compute remains.

AKS PVC now retains: qwen3:4b, qwen3:8b, qwen3:14b, lfm2.5:latest (21/30Gi used).

AKS results

small profile, live Ollama, deterministic 23-scenario pack:

model / run	result	observations
Qwen3 8B, full pack, D16 Spot	20/23 (87%)	malformed 0; safety 0; PVC, healthy restart, stuck rollout all pass
Qwen3 8B, target core 3	3/3	best quality/speed balance
Qwen3 4B, target 5	4/5	PVC answer missed explicit absence once; all evidence fetched
Qwen3 14B, target core 3	2/3 recorded	first cold load exceeded 60s; after load PVC+rollout pass, but PVC used 6 iterations/495s
LFM2.5, rollout (prewarmed)	0/1	fetched correct evidence with valid calls, final wording failed deterministic diagnosis assertion

Qwen3 8B full-pack failures: healthy-service-endpoints, image-pull-auth, service-selector-mismatch. These are the next measured targets; the previous three remaining failures are resolved in the AKS run.

Verification

• 3084 passed, 21 skipped
• mypy strict: clean (113 source files)
• tach: all modules validated
• full ruff + format check: clean
• independent code review findings all fixed in 16febdd

[hellices]

Consolidating this track into #176 as requested. The implementation history and measured prompt/tool findings here remain valid; all new conversational-eval design, model scores, and next actions will be maintained in #176.