Agent Training

A repository for designing, evaluating, and iterating AI agent training materials focused on MongoDB education.

📖 For detailed reference documentation, conventions, and architecture, see CLAUDE.md (authoritative source).

Quick Start

git clone https://github.com/danielcurran/agent-training
cd agent-training

Open in VS Code, enable "chat.promptFiles": true in settings, then type / in Copilot Chat to see available agents.

Repo Structure

agents/              → Agent definitions (role, task, behaviors, output format)
labs/
  outlines/          → High-level lab outlines
  specs/             → Detailed technical specs generated by the converter
  reports/           → Evaluation reports generated by the evaluator
  chunks/            → RAG-ready content chunks (concepts/ + tasks/ + manifest.json)
lab-test-env/        → Built lab environments (skeleton apps + check scripts)
skills/              → Custom Claude skills for specialized tasks
standards/           → Instructional design rulebook and research sources
.github/prompts/     → Copilot Chat prompt files (invoke with / in chat)

Agents

Agent	Purpose	Invoke with
Lab Outline Designer	Design new lab outlines	/design-lab-outline
Lab Outline Converter	Convert outlines into zero-knowledge technical specs	/convert-lab-outline
Lab Instruction Evaluator	Orchestrate both evaluators and combine scores	/evaluate-lab-instructions
Spec Quality Evaluator	Evaluate spec structure and rulebook compliance	/evaluate-spec-quality
Learner Experience Evaluator	Evaluate completion likelihood from zero-knowledge perspective	/evaluate-learner-experience
Lab Environment Builder	Generate skeleton app and check scripts from a tech spec	/build-lab-environment
Agent Learner	Complete a lab as an AI learner and produce a learning report	/run-learner-agent
Transfer Task Scorer	Score learner's transfer task; produce KLI hypothesis verdicts and hypothesis-validation finding	/score-transfer-task
RAG Chunker	Convert validated lab content into retrieval-ready chunks for LLMs	/chunk-lab-content
RAG Chunk Evaluator	Score chunks for standalone coherence and retrieval quality	/evaluate-lab-chunks

Workflow

1. Design a lab outline           → /design-lab-outline → labs/outlines/[name]-outline.md
2. Convert to tech spec           → /convert-lab-outline → labs/specs/[name]-tech-spec.md
3. Evaluate spec quality & learner experience → /evaluate-lab-instructions → labs/reports/[name]/[name]-tech-spec-eval-v{N}.md
4. Iterate spec on feedback       → repeat until both scores ≥ 8/10
5. Build lab environment          → /build-lab-environment → lab-test-env/[name]/
6. Run learner agent              → /run-learner-agent → labs/reports/[name]/[name]-env-eval-v{N}.md
7. Score transfer task            → /score-transfer-task → labs/reports/[name]/[name]-transfer-v{N}.md
8. Chunk lab content for RAG      → /chunk-lab-content → labs/chunks/[name]/ (concepts/ + tasks/ + manifest.json)
9. Evaluate chunk quality         → /evaluate-lab-chunks → labs/reports/[name]/[name]-chunk-eval-v{N}.md

Setup

1. Open this repo in VS Code
2. Ensure chat.promptFiles is enabled in settings: "chat.promptFiles": true
3. Open Copilot Chat and type / to see available prompts

Documentation

• CLAUDE.md — Comprehensive reference (architecture, conventions, full workflow)
• standards/instructional-design-rulebook.md — Pedagogical guidelines
• standards/hypothesis-validation.md — Lab-by-lab hypothesis testing results and revision log
• standards/sources/research-sources.md — Learning science research backing the rulebook
• skills/README.md — Custom Claude skills guide
• lab-execution/README.md — Lab environment template documentation

---

Changelog

May 2026 — Cross-Session Knowledge Retention

Added infrastructure for agents to retain what they learn between lab sessions. Previously, the learner agent completed a lab and produced a markdown report, but all knowledge was ephemeral — the next run started from scratch. This change introduces a structured knowledge artifact and validation layer across all production labs.

What changed:

• KNOWLEDGE.json required in all labs. Each lab now instructs the agent to produce a machine-readable knowledge file after completing stages. The file is a JSON array of knowledge entries, each capturing the concept learned, the SQL instinct it replaces, the rule in one sentence, when to apply it, and a confidence level grounded in whether the associated milestone check passed (verified), required correction (corrected), or was self-assessed. This gives every knowledge claim a provenance signal.

• check:knowledge added to all lab environments. Check scripts validate schema structure, minimum entry count, required fields, and lab-specific concept coverage (e.g., ESR lab must cover the equality-sort-range rule; memory-for-ai must cover namespace isolation and vector retrieval). The check runs as the final step of check:all. Labs: Builder Badge, ESR Indexing Strategy, Aggregation Foundations, Memory for AI.

• Learner agent updated. agents/learner.md now includes a Step 4 after the learning report: generate and validate KNOWLEDGE.json. The agent is given the full schema, confidence value definitions, and the validation command for both Node.js and Python lab environments.

• Rulebook updated. Rule 10 (Reflection and Decision Records) now includes the KNOWLEDGE.json requirement, full schema, and lab environment expectations for spec authors and the environment builder agent.

---

April–May 2026 — Instruqt Integration (Abandoned)

Investigated running the learner agent inside Instruqt browser-based lab environments. Attempted both API-token and Playwright browser automation approaches. Neither worked reliably — Instruqt's terminal DOM is not stable enough for programmatic automation without official API support. Code and documentation removed; local Docker remains the execution path.