A framework for AI-driven scientific discovery. Turn AI into a research partner — not just a code optimizer.
Autoresearch asks "did the metric improve?" Deep Research Agent asks "why — and what should we try next?"
Karpathy's autoresearch proved that AI agents can run hundreds of experiments overnight. AIDE added tree search over code. But Meta's MLGym research revealed a fundamental limitation: current AI agents can optimize hyperparameters, but they cannot generate novel hypotheses.
Blind optimization hits ceilings. When your agent exhausts the local search space, it starts making random changes with diminishing returns. It doesn't know why something works, so it can't reason about what to try next.
Research isn't a loop — it's a spiral. Each experiment should inform the next through accumulated domain knowledge, not just a scalar metric.
A student team with zero domain expertise used this framework to compete in a national AI competition (AI Cup 2026 — classifying bird species from radar data, 9 classes, 90 teams):
- 200+ experiments systematically tracked across 6 weeks
- AI autonomously acquired expertise in radar physics, ornithology, and sensor engineering
- Discovered that 21 of 40 Cormorant labels were incorrect through cross-referencing radar signatures with species body mass
- Built a custom validation framework when standard cross-validation failed to predict leaderboard performance
- Maintained a failure catalog of 20+ dead ends that prevented hundreds of hours of wasted effort
Result: 4th place overall. Not by having better models — the final model was standard LightGBM + CatBoost. By knowing which 36 features to keep, why post-processing works on unseen months, and when to stop exploring dead ends.
Four pillars that turn AI from a code generator into a research collaborator:
┌──────────────────────────────────────────┐
│ KNOWLEDGE ARCHITECTURE │
│ Three-tier persistent memory system │
│ Hot: project rules (CLAUDE.md) │
│ Warm: domain knowledge (memory files) │
│ Cold: research prompts (on-demand) │
└──────────────────┬───────────────────────┘
│
┌────────────────────────────┼────────────────────────────┐
│ │ │
▼ ▼ ▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ AGENT-FRIENDLY │ │ EXPERIMENT │ │ RESEARCH │
│ CODEBASE │ │ PROTOCOL │ │ LOOP │
│ │ │ │ │ │
│ Modular code │ │ Numbered scripts │ │ 1. Research │
│ that minimizes │ │ Shared src/ │ │ 2. Hypothesize │
│ hallucination │ │ Logged metrics │ │ 3. Experiment │
│ and context │ │ Failure catalog │ │ 4. Analyze │
│ bloat │ │ │ │ 5. Learn │
└──────────────────┘ └──────────────────┘ └──────────────────┘
A three-tier memory system — inspired by the Codified Context paper (one developer, 108K lines of code, 70 days) — that ensures the AI never starts from zero:
| Tier | What | When Loaded | Purpose |
|---|---|---|---|
| Hot | Project rules (CLAUDE.md) |
Every session | Constraints, DO NOT RETRY list, domain caveats |
| Warm | Domain knowledge (memory files) | Relevant sessions | Accumulated research, species signatures, thresholds |
| Cold | Research prompts | When researching | Templates for literature review, data audit, evidence discovery |
Knowledge flows upward: a research finding becomes domain knowledge, which becomes a project rule.
Code architecture directly affects AI agent performance. CodeScene research found that AI on unhealthy code produces 30% more defects and consumes 2x the tokens compared to modular codebases. SWE-bench showed that providing tests doubles agent success rates.
Five principles:
- Semantic modules — one concern per file, named by function
- Small context surface — 200-400 lines per file, 800 max
- Composable functions — experiments compose extractors, no inheritance chains
- Single source of truth — metrics, data loading, constants in one place
- No implicit state — data in, results out
Every experiment is a unit of scientific work with a hypothesis, a result, and a lesson:
- Numbered scripts (
e001_baseline.py,e002_add_rcs_features.py) inexperiments/ - Shared source library (
src/) — never copy-paste core logic - Structured log (
EXPERIMENTS.md) — ID, date, hypothesis, all metrics, outcome - Failure catalog — dead ends documented with why, not just what
The five-phase cycle that connects knowledge to action:
RESEARCH ──→ HYPOTHESIZE ──→ EXPERIMENT ──→ ANALYZE ──→ LEARN
▲ │
└──────────────── knowledge base ◄─────────────────────┘
| Phase | What the AI Does | Output |
|---|---|---|
| Research | Synthesizes papers, discovers data sources, learns domain physics | Domain knowledge docs |
| Hypothesize | Proposes experiments grounded in domain knowledge + past failures | Testable hypothesis with rationale |
| Experiment | Implements and runs using templates and shared library | Metrics + artifacts |
| Analyze | Per-class breakdown, error analysis, confusion patterns | Understanding of why |
| Learn | Updates knowledge base, adds to failure catalog, refines rules | Cumulative intelligence |
Phases 1, 2, and 5 don't exist in autoresearch. That's where the strategic advantage comes from.
| Autoresearch | AIDE | AI Scientist | Deep Research Agent | |
|---|---|---|---|---|
| Search strategy | Linear hill-climb | Tree search | Template-guided | Hypothesis-driven |
| Domain knowledge | None | None | Literature (automated) | Deep, accumulated, persistent |
| Scope | 1 file, 1 metric | Multi-file, 1 metric | Paper output | Full codebase, multi-metric |
| Novel hypotheses | No | No | Limited | Yes (via domain research) |
| Failure memory | Git revert | Tree pruning | None | Cataloged with reasons |
| Knowledge persistence | results.tsv | Solution tree | Papers | Three-tier memory system |
| Human role | Sleep | Configure | Review papers | Steer research direction |
# 1. Copy the framework templates into your project
cp -r framework/1-knowledge-architecture/CLAUDE.md.template your-project/CLAUDE.md
cp -r framework/1-knowledge-architecture/memory/ your-project/memory/
cp -r framework/3-experiment-protocol/experiment-template.py your-project/experiments/
cp -r framework/3-experiment-protocol/EXPERIMENTS.md.template your-project/EXPERIMENTS.md
# 2. Fill in your domain in CLAUDE.md
# Replace [YOUR DOMAIN], [YOUR METRIC], [YOUR CLASSES] placeholders
# 3. Start the research loop — first session prompt:
# "Research the domain of [X]. What are the key physics/mechanics?
# What signals discriminate between classes? Document in memory."The framework was born from a real competition. Read how AI went from zero ornithology knowledge to discovering mislabeled data, building custom validation systems, and placing 4th out of 90 teams.
- Autoresearch integration — use blind optimization as a sub-routine of the experiment phase
- Multi-agent research teams — specialized agents for research, hypothesis generation, error analysis
- Domain templates — plug-in research prompts for biology, finance, materials science, NLP
- Hypothesis scoring — rank experiments by expected information gain from the knowledge base
- Cross-project knowledge transfer — meta-learnings that transfer across problem domains
- Community case studies — prove the framework generalizes beyond bird radar
This framework draws on and is validated by:
- Karpathy, A. (2026). autoresearch. 700 autonomous experiments, 20 genuine improvements.
- Jiang et al. (2025). AIDE: ML Engineering with Tree Search. 4x more Kaggle medals than linear agents.
- Nathani et al. (2025). MLGym: Gym for ML Research. Agents optimize hyperparams but don't generate novel hypotheses.
- Author redacted (2026). Codified Context: Infrastructure for AI Agents. 108K-line system, 1 developer, 70 days, 24% knowledge infrastructure.
- Borg & Tornhill (2025). Code for Machines, Not Just Humans. 30% more defects on unhealthy code.
- Yang et al. (2024). SWE-agent. Agent-Computer Interface design improves resolve rates by 10.7pp.
- Yamada et al. (2025). AI Scientist v2. First AI paper accepted through peer review.
MIT
This framework is young. If you use it on a problem and it works (or doesn't), we want to hear about it. Open an issue or submit a case study.