Public-safe measurement notes from a private sealed evaluation lab for local coding agents, constrained generation, RAG routes, skill/instruction docs, and live completion-honesty probes.
Author: Cameron Sanderson
This package is reports, essay, and method language only. It does not include private sealed fixtures, raw agent transcripts, or full lab code.
| Surface | URL |
|---|---|
| Guided tour (GitHub Pages) | https://camerontjs-dot.github.io/agent-eval-notes/ |
| This repo | https://github.com/camerontjs-dot/agent-eval-notes |
| Runnable honesty demo | https://github.com/camerontjs-dot/verified-done |
Tour order: Start → Methodology → Labels → H1 rejection → Essay → Verify ablation → Multi-path → Transfer → RAG → Skills → Try the method → Limitations → Resume bullets.
Local preview:
python3 -m http.server 8080 --directory docs| Surface | Role |
|---|---|
| This repo + Pages tour | Methods, locked numbers, harness disclosure, essay |
| verified-done | Runnable honesty demo (tasks + selftest + live evidence summary) |
| Related apparatus | claim-audit-lab, evidence-bundler, apparatus-contracts |
| Workspace OS | MainFrame (public Stage 1b cut; MindGraph nested) |
Operator publish logs and internal program notes stay in the private workspace. They are not part of this repo.
| Path | Contents |
|---|---|
docs/ |
GitHub Pages guided tour |
reports/ |
Six self-contained Markdown write-ups (01–06) |
essays/ |
Longform H1 rejection essay |
briefs/ |
One-pagers, harness isolation protocol, public harness descriptions |
distribution/ |
LinkedIn paste body + post log |
pdf/ |
Print-ready PDFs (H1, multi-path, transfer) |
numbers-lock.md |
Locked headline numbers + cross-repo links |
METHODOLOGY.md |
How measurement is defined |
LIMITATIONS.md |
What these notes do not claim |
proof-points.md |
Resume / interview talking points (numbers-locked) |
distribution/linkedin-post-draft.md |
Ready-to-post LinkedIn draft |
notes/rag-adversarial-next.md |
RAG harder-slice plan (no new numbers yet) |
| Report | One-line story | Best for |
|---|---|---|
| 01 - H1 rejection | Better aggregate still not promoted | AI safety / eval |
| 02 - Multi-path coding | Multi-file is the discriminating case | Local coding agents |
| 03 - Task-family transfer | Coding winner fails constrained prose | Eval design |
| 04 - RAG routes | Use-case routes, not one Elo | RAG / retrieval |
| 05 - Agent-as-evaluator | Skill edge matrix + thin smoke receipts | Prompt / skills |
| 06 - Verify tool ablation | run_verify flips honesty (Haiku + local coder-14b); abstainer control | Agent reliability |
Essay: The harness that scored better and was not promoted
- Say measured / exploratory, not validated or proven, unless a confirmatory protocol is named.
- State n with every headline metric.
- Say not promoted, not thrown out, for the H1 decision.
- Name the stack (local open-weight vs live frontier) when it matters.
- DEV routing only unless a report documents a graduated path.
- Do not blend coding, prose, RAG, and skill scores into one model quality number.
- verified-done — runnable honesty demo split
- claim-audit-lab
- apparatus-contracts
- research-scaffold-harness
- evidence-bundler