RASE-Bench (Recursive Agent Skill Evolution Benchmark) is a principled, open-source framework for measuring and driving the recursive self-improvement of LLM agent reasoning strategies. Unlike static benchmarks that evaluate model capabilities at a single point in time, RASE-Bench operationalizes a core AGI hypothesis: an agent capable of improving its own thinking strategies, given sufficient compute and feedback, can bootstrap increasingly effective reasoning without human intervention.
The benchmark implements a closed-loop evolution pipeline where agents:
- Execute tasks using current skill templates
- Analyze their own reasoning trajectories to identify failure modes
- Propose, debate, and validate improved skill candidates
- Replace inferior skills with demonstrably better ones
- (Optionally) Meta-evolve the evolution prompt itself
All of this runs on consumer-grade hardware (RTX 3060, Apple M1, CPU-only) using quantized 7B-14B models — democratizing recursive self-improvement research.
Generation 0: baseline_score=0.42
↓ (analyze trajectories → propose skill → validate → keep if better)
Generation 1: score=0.51 (improvement: +0.09)
↓ (meta-evolution improves the evolution prompt itself)
Generation 2: score=0.58 (improvement: +0.07)
↓
... recursive improvement until convergence or budget exhausted
The ability to recursively self-improve is widely considered a critical capability on the path to Artificial General Intelligence (AGI). As noted by Schmidhuber (2007), Gödel machines can self-rewrite their own code in provably optimal ways. Modern LLM agents approximate this through skill evolution — rewriting the prompts and reasoning templates that govern their behavior.
RASE-Bench provides the first standardized testbed for measuring:
| AGI Capability | How RASE-Bench Measures It | Why It Matters |
|---|---|---|
| Recursive Self-Improvement | Score deltas across generations of skill evolution | An AGI must improve its own algorithms without human oversight |
| Meta-Cognition | Meta-evolution layer improves the evolution prompt itself | The system must reason about how it reasons |
| Collective Intelligence | Multi-agent debate generates better skill proposals than individuals | Super-human alignment and discovery may require agent collectives |
| Resource-Efficient Improvement | Tokens-per-improvement, wall-clock per generation | Practical AGI must improve efficiently, not just effectively |
| Out-of-Distribution Generalization | Held-out task validation after each generation | True improvement transcends the training distribution |
| Convergence Dynamics | Plateau detection, improvement decay curves | Understanding when and why self-improvement stalls is critical for safety |
Without standardized benchmarks for recursive self-improvement, the field cannot systematically study what may be the most important capability to measure in near-term AI systems.
RASE-Bench builds on and differs from several key research threads:
| Work | Relation to RASE-Bench | Key Difference |
|---|---|---|
| Gödel Machines (Schmidhuber, 2007) | Theoretical foundation for self-rewriting agents | RASE-Bench provides an empirical, practical implementation |
| Self-Improving GPT (Huang et al., 2022) | LLMs generate self-improvement data | RASE-Bench focuses on skill (prompt/reasoning strategy) evolution, not fine-tuning |
| MetaSkill-Evolve (Zhang et al., 2024) | Evolves agent skills via LLM proposals | Requires cloud GPUs; no held-out validation; no meta-evolution |
| DSPy (Khattab et al., 2023) | Programmatic prompt optimization | No recursive evolution; no collective debate; no meta-level improvement |
| Recursive Agent Harness (Shinn et al., 2024) | Agent reflection and refinement | Partial meta-cognition; no parallel debate; no held-out tasks |
| STaR (Zelikman et al., 2022) | Self-taught reasoning via rationalization | Generates training data, not evolvable skill templates |
| Constitutional AI (Bai et al., 2022) | Self-critique and revision of LLM outputs | RASE-Bench uses similar critique-but for skills, not outputs |
| Voyager (Wang et al., 2023) | Open-ended skill discovery in Minecraft | RASE-Bench provides structured, reproducible evaluation across diverse task categories |
| Held-Out Validation (Kuhn et al., 2023) | Preventing overfitting in self-improving systems | RASE-Bench integrates held-out tasks as a core architectural constraint |
| Approach | Local HW? | Meta-Evolution? | Collective Debate? | Efficiency Tracking? | Held-Out? | Open Source? |
|---|---|---|---|---|---|---|
| RASE-Bench | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes |
| MetaSkill-Evolve | ❌ Cloud GPUs | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
| Recursive Agent Harness | ❌ Cloud APIs | ❌ No | ||||
| DSPy | ✅ Yes | ❌ No | ❌ No | ❌ No | ✅ Yes | |
| Voyager | ❌ Cloud APIs | ❌ No | ❌ No | ❌ No | ❌ No | ✅ Yes |
flowchart TD
Start([Start]) --> Gen0[Gen 0: Run Tasks<br/>Collect Baseline]
Gen0 --> Traj[Collect Reasoning<br/>Trajectories]
Traj --> Debate[Collective Debate:<br/>Multiple Agents<br/>Critique & Propose]
Debate --> Evolve[Evolution Engine:<br/>Generate New Skill<br/>Candidate]
Evolve --> Run[Run Tasks with<br/>New Skill Candidate]
Run --> Validate[Held-Out<br/>Validation]
Validate --> Improve{Improvement?}
Improve -->|Yes| Keep[Accept Skill &<br/>Record Metrics]
Improve -->|No| Reject[Reject Skill,<br/>Retain Previous]
Keep --> Meta{Meta-Evolution<br/>Trigger?}
Meta -->|Yes| MetaEvolve[Improve the<br/>Evolution Prompt]
MetaEvolve --> Check
Meta -->|No| Check{Budget<br/>Exhausted?}
Reject --> Check
Check -->|No| Traj
Check -->|Yes| Report[Report Final Metrics<br/>Generalization Scores]
Report --> Stop([Stop])
┌─────────────────────────────────────────────────────────────────────┐
│ RASE-Bench Main Loop │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Gen 0: Run tasks → baseline metrics │
│ ↓ │
│ For gen in 1..N: │
│ ├─ 1. Collect trajectories from last run │
│ ├─ 2. Run Collective Debate on skill proposals │
│ ├─ 3. Evolution Engine → new skill candidate │
│ ├─ 4. Run tasks with new skill │
│ ├─ 5. Validate on held-out tasks │
│ ├─ 6. Track improvement metrics │
│ └─ 7. (Every k gens) Meta-Evolution: improve evolve prompt │
│ ↓ │
│ Report: scores, efficiency, generalization │
│ │
└─────────────────────────────────────────────────────────────────────┘
RASE-bench/
├── README.md # This file
├── pyproject.toml # Package config
├── requirements.txt # Dependencies
├── scripts/
│ ├── run_benchmark.py # Main entry point
│ └── analyze_results.py # Results analysis & visualization
├── rase_bench/
│ ├── __init__.py # Package init
│ ├── config.py # BenchmarkConfig dataclass
│ ├── utils.py # Logging, I/O, timing utilities
│ ├── agent_harness.py # LLM backend (Ollama/llama.cpp)
│ ├── task_suite.py # Task loading & train/held-out split
│ ├── evolution_engine.py # Skill evolution from trajectories
│ ├── collective_debate.py # Multi-agent debate module
│ ├── meta_evolution.py # Improve-the-improver loop
│ ├── held_out_validator.py # Generalization checking
│ ├── metrics.py # Data classes for tracking
│ ├── benchmark_runner.py # Orchestrator / main loop
│ └── tasks/
│ ├── __init__.py
│ ├── base_task.py # Abstract task class
│ ├── coding_optimization.py
│ ├── trajectory_analysis.py
│ ├── experiment_design.py
│ └── prompt_engineering.py
├── evolved_skills/ # Saved evolved skill candidates
├── results/ # Benchmark run output
│ └── run_YYYYMMDD_HHMMSS/
│ ├── summary.json
│ ├── generation_0.json
│ ├── generation_1.json
│ └── ...
└── examples/
└── example_run.sh
| Category | Tasks | Evaluation Metric | Relevance to AGI |
|---|---|---|---|
| Coding Optimization | Optimize sum-of-squares, vectorize sliding window | Correctness + speedup | Autonomous code improvement is a prerequisite for self-modifying systems |
| Trajectory Analysis | Analyze reasoning trace, identify redundant steps | Key concept F1, redundancy detection | Meta-cognition requires understanding one's own reasoning |
| Experiment Design | Design ablation study, propose evaluation metric | Completeness rubric (5 criteria) | Autonomous research requires formulating rigorous experiments |
| Prompt Engineering | Improve CoT prompt, design system prompt | Structural quality, clarity, robustness | Prompt engineering is the primary interface for skill evolution |
| 🧪 Future: Safety & Alignment | Red-teaming, value learning probes | Harm detection, value alignment | Recursive improvement must be steered toward safe outcomes |
- Python 3.10+
- One of:
- Ollama (recommended) —
curl -fsSL https://ollama.com/install.sh | sh - llama.cpp server running at
http://127.0.0.1:8080
- Ollama (recommended) —
git clone https://github.com/NullLabTests/RASE-bench.git
cd RASE-bench
pip install pyyamlollama pull qwen2.5:7b # Recommended: 7B for consumer GPUs
ollama pull qwen2.5:3b # Smaller, faster
ollama pull qwen2.5:7b-q4_K_M # CPU-friendly quantizedpython scripts/run_benchmark.py --dry-run --generations 3python scripts/run_benchmark.py --model qwen2.5:7b --generations 10python scripts/analyze_results.py results/run_YYYYMMDD_HHMMSS/ --details| Argument | Default | Description |
|---|---|---|
--backend |
ollama |
LLM backend: ollama, llamacpp, or dry-run |
--model |
qwen2.5:7b |
Ollama model tag or GGUF name |
--gguf-path |
None | Path to GGUF file (llamacpp backend) |
--generations |
10 | Number of evolution generations |
--population |
4 | Number of skill candidates to keep per generation |
--debate-agents |
3 | Number of agents in collective debate |
--gpu-layers |
20 | GPU offload layers (0 = CPU only) |
--no-meta |
False | Disable meta-evolution |
--log-dir |
results |
Directory for results output |
--dry-run |
False | Use simulated LLM (no backend needed) |
--seed |
42 | Random seed |
Each run produces a timestamped directory under results/run_YYYYMMDD_HHMMSS/:
summary.json — Compact run overview:
{
"model_name": "qwen2.5:7b",
"total_tokens": 152340,
"duration_seconds": 1234.5,
"final_best_score": 0.723,
"improvement_over_baseline": 0.251,
"generation_scores": [0.472, 0.511, 0.568, 0.612, 0.654, 0.689, 0.723],
"generation_success_rates": [0.375, 0.500, 0.500, 0.625, 0.750, 0.750, 0.875]
}generation_N.json — Detailed per-generation metrics including per-task scores, tokens, attempts, held-out validation results, skill proposal details, and improvement deltas.
The analyze_results.py script generates:
- 📈 Score trajectories across generations (with improvement arrows)
- 💰 Token efficiency (score gain per 10K tokens)
- 🧪 Generalization gap (train vs. held-out performance)
- ⏱️ Convergence detection (when improvement plateaus)
- 📊 Debate quality metrics (how often the debated skill outperforms the baseline)
| Model | GPU | Generations | Time | Total Tokens | Est. Improvement |
|---|---|---|---|---|---|
| Qwen 2.5 7B (Q4_K_M) | RTX 3060 12GB | 10 | ~45 min | ~150K | +0.20–0.35 |
| Qwen 2.5 7B (Q4_K_M) | CPU only (16 cores) | 5 | ~60 min | ~75K | +0.10–0.20 |
| Qwen 2.5 3B (Q4_K_M) | RTX 3060 12GB | 10 | ~15 min | ~120K | +0.15–0.25 |
| Llama 3.2 3B | Apple M1 8GB | 10 | ~20 min | ~100K | +0.15–0.25 |
Times are approximate. Improvement ranges are preliminary and depend on task complexity, debate rounds, and model capability.
Extending RASE-Bench with custom tasks is straightforward:
- Create a new file in
rase_bench/tasks/(e.g.,my_task.py) - Subclass
BaseTaskand implementget_prompt()andevaluate() - Register it in
rase_bench/task_suite.py
from .base_task import BaseTask, TaskResult
class MyNewTask(BaseTask):
def __init__(self):
super().__init__(
task_id="my_001",
name="Descriptive Name",
description="What the agent needs to do",
category="my_category",
)
def get_prompt(self) -> str:
return "Your task prompt here..."
def evaluate(self, response: str, trajectory) -> TaskResult:
# Score response, return TaskResult(success, score, feedback)
...Recursive self-improvement research carries inherent risks. RASE-Bench incorporates several safety-by-design principles:
- Bounded Evolution — Generations are capped (default: 10). Improvement is constrained to skill templates, not model weights or system prompts.
- Held-Out Validation — Prevents deceptive overfitting where a skill "games" the evaluation metric without genuine improvement.
- Full Traceability — Every generation logs complete trajectories, proposals, debate transcripts, and validation results for audit.
- Local-Only Execution — All computation runs on local hardware. No data leaves the user's machine.
- Dry-Run Mode — Enables testing the pipeline and code without any LLM interaction.
We strongly recommend:
- Reviewing all evolved skills before deployment in any production system
- Using the
--generationsflag to limit compute budget - Running held-out validation on tasks completely unrelated to the training set
If you use RASE-Bench in your research, please cite:
@software{rase_bench_2025,
author = {Null Labs and RASE-Bench Contributors},
title = {{RASE-Bench}: Recursive Agent Skill Evolution Benchmark},
year = {2025},
publisher = {GitHub},
url = {https://github.com/NullLabTests/RASE-bench},
doi = {10.xxxx/xxxxx},
note = {Local edition. arXiv:XXXX.XXXXX}
}@article{schmidhuber2007godel,
title={G{\"o}del machines: Fully self-referential optimal universal self-improvers},
author={Schmidhuber, J{\"u}rgen},
journal={Artificial General Intelligence},
year={2007},
url={https://arxiv.org/abs/0712.2209}
}
@article{wang2023voyager,
title={Voyager: An Open-Ended Embodied Agent with Large Language Models},
author={Wang, Guanzhi and Xie, Yuqi and Jiang, Yunfan and others},
journal={arXiv preprint arXiv:2305.16291},
year={2023}
}
@article{khattab2023dspy,
title={DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines},
author={Khattab, Omar and others},
journal={arXiv preprint arXiv:2310.03714},
year={2023}
}
@article{bai2022constitutional,
title={Constitutional AI: Harmlessness from AI Feedback},
author={Bai, Yuntao and others},
journal={arXiv preprint arXiv:2212.08073},
year={2022}
}
@article{zelikman2022star,
title={STaR: Bootstrapping Reasoning With Reasoning},
author={Zelikman, Eric and Wu, Yuhuai and Goodman, Noah D},
journal={arXiv preprint arXiv:2203.14465},
year={2022}
}We welcome contributions from the research community! Areas where help is especially valuable:
| Area | How to Help |
|---|---|
| New Tasks | Create tasks that test relevant agent capabilities (safety, planning, tool use, etc.) |
| Improved Evaluation | Make scoring more nuanced and reliable (rubric-based, pairwise, LLM-as-judge) |
| New Backends | Add support for Anthropic, OpenAI, or other local inference engines (vLLM, TensorRT-LLM) |
| Visualization | Better result analysis, interactive plots, paper-ready figures |
| Benchmarking | Run on different models/hardware and contribute results to the community table |
| Theory | Formalize convergence bounds, sample complexity of skill evolution |
See CONTRIBUTING.md for detailed guidelines.
MIT — See LICENSE for full details.
If RASE-Bench accelerates your research, please:
- ⭐ Star the repository on GitHub
- 📄 Cite the paper using the bibtex above
- 🐛 Open issues for bugs or feature requests
- 💬 Start discussions for research questions or collaboration
Built by researchers, for researchers. Democratizing recursive self-improvement.