A research and architecture repository for eval-guided improvement loops.
The core idea is simple:
system variant -> eval -> score/trace -> patch or mutate -> archive -> gate -> repeat
This can run in two modes:
Mode A: Config / Prompt Evolution
AI app config -> TaskAdapter -> eval harness -> mutator -> better config
Mode B: Coding Agent Patch Loop
repo snapshot -> coding agent -> code patch -> tests/benchmark -> rollback or promote
The governing thesis: agents are replaceable, evals are the control plane. Agents, scaffolds, and models are disposable variation producers; the eval system — specs, hidden tests, regression oracle, archive, promotion/rollback — is the durable selection layer and the only part of the stack that gains value across model generations. Target shape: framework-light, eval-heavy, adapter-based, agent-agnostic. The full argument and the three-layer architecture live in Golden_Quality_Setup.md § Architecture Thesis.
This repository is currently an architecture and research package. It does not yet contain runnable implementation code.
Read these first:
-
Golden_Quality_Setup.md
The target architecture, implementation order, eval gates, and coding-agent patch-loop additions. -
Generisches_Eval_Harness_Framework.md
The main framework document, now including section 10 on coding-agent patch mode. -
EvalSpec_Guide.md
The portable spec format for generating QA cases, golden datasets, component checks, trace checks, and guardrails. -
Dataset_Quality_Checklist.md
The audit checklist for deciding whether a generated eval dataset is actually trustworthy. -
Continual_Learning_Loop.md The production failure cluster -> hypothesis -> eval case -> gated fix loop.
-
research/2026-07-07-agentic-eval-runtime/report.md The sourced research report behind the generic eval/runtime architecture.
For coding-agent self-improvement, read this too:
This repo explores a generic runtime for improving AI systems without rebuilding the eval loop for every project.
The runtime has two related jobs.
Use this when the target system is an AI app whose behavior can be improved by changing prompts, configs, retrieval parameters, routing choices, model choices, or similar runtime settings.
Examples:
- RAG systems
- classifiers
- chatbots
- tool-using agents
The app implements a TaskAdapter. The eval loop, archive, mutator, convergence logic, and guardrails remain reusable.
Use this when the target system is a codebase and a coding agent is allowed to produce patches until a benchmark, specification, or baseline gate is reached.
Examples:
- fixing project-local regression suites
- resolving SWE-bench-style issues
- improving a coding-agent scaffold
- comparing candidate agent/tooling variants
The eval system is the judge. The coding agent is the patch producer.
The current best setup is:
TaskAdapter
+ DeepEval / Inspect runner
+ RAGAS / ARES metrics for RAG
+ DSPy / GEPA-style optimizer
+ immutable guardrail layer
+ event-sourced archive
+ strict train / validation / holdout / redteam split
In plain terms:
- own the adapter/runtime boundary
- reuse mature eval tools where possible
- keep guardrails outside the mutation space
- store every candidate and score in an archive
- optimize on visible splits only
- deploy only after holdout and redteam gates pass
For coding agents, add:
Task / Spec
-> Repo Snapshot
-> Coding Agent
-> Patch
-> Unit Tests
-> Benchmark / Regression Oracle
-> Promote or Roll Back
-> Archive Lineage
For coding agents, the runtime has a second mode:
Benchmark / Spec
-> Coding Agent proposes code patch
-> Unit tests gate the patch
-> Benchmark eval compares against best snapshot
-> Regression triggers rollback
-> Improvement updates best snapshot
-> Archive records lineage
Key distinction:
- the eval system is the judge, benchmark, archive, and rollback controller
- the coding agent is the patch producer
- the guardrail layer protects evaluator, hidden tests, archive, and safety rules from modification
The agent may patch application code or, in an advanced mode, its own scaffold. It must not patch the evaluator, hidden tests, guardrails, or archive.
Evidence strength behind this mode, so it isn't read as more proven than it is:
| Reference | Status |
|---|---|
| SWE-bench, SWE-agent | solid — replicated, widely used benchmark substrate |
| DSPy, GEPA | solid — mature, production-used optimizers |
| Darwin Godel Machine | directional — real results, but extreme compute cost |
| Self-Harness | directional — most concrete loop mechanics (editable surfaces, causal weakness mining, dual acceptance), tested across three base models |
| TDAD | speculative — single 2026 preprint, 10-instance subset for its headline number |
| Kitchen Loop | speculative — single 2026 preprint, unreplicated |
The architecture's core rule — evaluator and permission control live outside the loop that evolves the system — is independently endorsed by Lilian Weng's harness-engineering survey (Jul 2026).
Full safeguards, phasing, and the archive/rollback design live in Golden_Quality_Setup.md § Coding Agent Patch Mode — treated there as deferred, contingent on Mode A proving ROI, not a parallel effort.
When an agent works in this repo:
- Read Golden_Quality_Setup.md first.
- Read Generisches_Eval_Harness_Framework.md for the original design intent.
- Read EvalSpec_Guide.md before generating QA cases, golden datasets, or benchmark tasks.
- Use Dataset_Quality_Checklist.md before calling a dataset good.
- Use Continual_Learning_Loop.md when turning production traces, logs, or support failures into eval-backed improvements.
- Use research/2026-07-07-agentic-eval-runtime/report.md before making claims about generic eval/runtime tools.
- Use research/2026-07-07-coding-agent-patch-loop/report.md before making claims about self-improving coding agents.
- Use research/2026-07-07-eval-dataset-quality/report.md before making claims about dataset quality.
- Treat guardrail isolation as non-negotiable.
- Do not implement a generic eval framework from scratch without checking whether DeepEval, Inspect AI, RAGAS, ARES, DSPy, GEPA, promptfoo, SWE-bench, TDAD, or SICA already cover the need.
- If adding implementation code, keep the first version adapter-first and small.
Useful first implementation target:
src/
agentic_eval_runtime/
adapters.py
cases.py
metrics.py
archive.py
mutator.py
runner.py
reports.py
Suggested first CLI:
aer eval --adapter path.to.Adapter --config config.yaml --dataset datasets/validation.jsonl
aer mutate --adapter path.to.Adapter --config config.yaml --dataset datasets/train.jsonl
aer report --archive runs/latest/archive.sqlite
Suggested coding-agent CLI:
aer patch-loop --repo path/to/repo --task tasks/001.yaml --baseline baseline.yaml
aer promote --archive runs/latest/archive.sqlite --candidate cand_042
aer rollback --archive runs/latest/archive.sqlite --to-best
Use this repo as a decision guide before implementation.
Best reading path:
- Read the Golden Quality Setup for the target design.
- Skim the framework concept to understand the abstraction.
- Read the EvalSpec guide to understand how QA cases and datasets are generated.
- Read the Dataset Quality Checklist to understand when a dataset is trustworthy.
- Use the generic research report to check why the recommended eval tools were chosen.
- Use the coding-agent research report for TDAD, SICA, DGM, HGM, SWE-bench, and Kitchen Loop context.
Questions this repo answers:
- What should we build ourselves?
- What should we reuse?
- How should a golden dataset be structured?
- How should optimizer-visible data be separated from holdout data?
- How do we prevent the mutation loop from removing safety checks?
- What should the first implementation milestone be?
- How can a coding agent patch toward a benchmark baseline without gaming the evaluator?
Each app implements a small adapter. Keep the v1 contract to four methods; do not add get_layer_boundaries() or get_coupling_constraints() until a real coupling constraint shows up in the first adapter. Designing those two in the abstract, before a second app exists to validate the shape, is how generic layers end up wrong.
class TaskAdapter(Protocol):
def run(self, case: EvalCase, config: dict) -> RunResult:
...
def get_config_schema(self) -> dict:
...
def validate_config(self, config: dict) -> list[ConstraintViolation]:
...
def extract_trace(self, result: RunResult) -> dict:
...The adapter is the only part that should be rewritten for each AI application. Layer boundaries and coupling constraints belong on this contract eventually — add them the moment the MUCi adapter needs one, not before.
extract_trace() returns traces in OpenTelemetry GenAI semantic-convention shape — vendor-neutral, so production and eval share one trace format. Rationale, tooling recommendation (Langfuse self-hosted for production, none/Phoenix for dev), and the production privacy policy (metadata-only by default, content capture opt-in): Golden_Quality_Setup.md § Trace Format.
Use four sets:
train.jsonl visible to optimizer
validation.jsonl used for candidate selection
holdout.jsonl never visible to mutator
redteam.jsonl adversarial and regression gate
The mutation loop must never see holdout or hidden redteam cases.
Run hard gates before scoring.
Hard disqualifiers include:
- guardrail violation
- forbidden claim
- invalid output schema
- missing required citation
- violated coupling constraint
- unsafe tool permission
- holdout leakage
Only candidates that pass hard gates should receive a composite score.
The mutator gets failure examples, traces, scores, and constraints. It does not get guardrail definitions, holdout cases, hidden redteam cases, or deployment thresholds.
This separation is critical because an optimizer can discover that removing a safety check improves score or latency.
Every candidate should be archived, including failures.
Archive fields should include:
- full config
- config patch
- parent candidate
- mutation hypothesis
- scores
- hard-gate result
- trace summary
- failure examples
- cost
- latency
- model versions
- timestamp
For coding agents, the archive must also capture patch lineage:
- task id
- repo snapshot
- patch/diff
- changed files
- commands run
- visible test output
- hidden test summary
- benchmark score
- regression count
- rollback reason
- parent snapshot
- best snapshot at evaluation time
Baseline success should require both resolution and non-regression.
Do not rebuild these from scratch unless there is a clear reason:
- general eval runner: DeepEval or Inspect AI
- RAG metrics: RAGAS / ARES-style metrics
- prompt/module optimization: DSPy
- reflective prompt/config evolution: GEPA-style loop
- prompt/provider regression and red-team checks: promptfoo
Build these locally:
- app-specific adapters
- coupling constraints
- guardrail isolation
- archive schema
- convergence logic
- final release gate
- reports tailored to this runtime
The first useful implementation is a single sequential path, not two parallel tracks. Generic interfaces are cheap to design and expensive to un-wrongly-generalize, so the plan below delays them until a manual baseline has proven the mutation loop is worth automating.
- Build the golden dataset for one real RAG/MUCi adapter: train, validation, holdout, redteam splits, with real labeled cases, not placeholders.
- Implement that one adapter directly against a small eval script. Do not build the generic
TaskAdapter/Archiveinterfaces yet. - Add deterministic metrics and hard guardrail disqualification before scoring.
- Run a manual mutation baseline: an engineer reads failure traces, edits the config/prompt by hand, re-runs, and records cost and iteration count in the archive (JSONL is enough).
- Only after step 4 exists, build the GEPA-lite mutator, and require it to beat the manual baseline on cost per point of validation-score improvement. If it doesn't, the mutator is not worth maintaining yet — say so and stop.
- Extract typed interfaces (
EvalCase,RunResult,Metric,TaskAdapter,Archive) only once a second real adapter (non-RAG) needs them. An abstraction extracted from one example is usually wrong.
Do not start with a dashboard. A CLI plus reproducible archive is the right first shape.
Treat this as a separate, later effort, not a parallel track. It carries higher engineering cost (sandboxing, snapshot management, evaluator protection) and thinner evidence than Mode A — see the evidence tags in Coding Agent Patch Mode above. Only start it once Mode A has shipped and proven the archive/gate/rollback pattern on the simpler config-mutation case.
- task spec with repo snapshot and expected baseline
- sandboxed coding-agent runner
- protected evaluator and hidden tests, SHA-256 checksummed
- unit-test gate before benchmark eval, rollback on failure or regression
- best-snapshot promotion and archive of patches, logs, scores, regressions, lineage
- specification/regression oracle inspired by Kitchen Loop
Primary research artifact:
Key external references:
- DeepEval: https://github.com/confident-ai/deepeval
- Inspect AI: https://github.com/UKGovernmentBEIS/inspect_ai
- promptfoo: https://github.com/promptfoo/promptfoo
- DSPy: https://github.com/stanfordnlp/dspy
- GEPA: https://github.com/gepa-ai/gepa
- OPRO: https://arxiv.org/abs/2309.03409
- TextGrad: https://arxiv.org/abs/2406.07496
- Agentic Benchmark Checklist: https://arxiv.org/html/2507.02825v2
- TDAD auto-improvement loop: https://arxiv.org/html/2603.17973v1
- SICA: https://arxiv.org/html/2504.15228v2
- Huxley-Godel Machine: https://arxiv.org/abs/2510.21614
- Kitchen Loop: https://arxiv.org/abs/2603.25697
- Weng, "Harness Engineering for Self-Improvement" (Jul 2026): https://lilianweng.github.io/posts/2026-07-04-harness/
- Self-Harness: https://arxiv.org/abs/2606.09498
- STOP: https://arxiv.org/abs/2310.02304
- ShinkaEvolve: https://arxiv.org/abs/2509.19349
This repository currently contains:
- framework concept
- architecture image
- generic eval/runtime research artifact
- coding-agent patch-loop research artifact
- eval dataset quality research artifact
- recommended golden setup
- this README
It does not yet contain implementation code, package metadata, tests, or runnable examples.