autoresearch: Detailed Introduction

[ikaijua]

autoresearch: Detailed Introduction

A small research environment where an existing agent repeatedly changes training code and compares experiments.

• Audience: People familiar with model training who have suitable GPU resources and want automated experiments.
• Cost: MIT code; GPU, storage, and the external agent’s inference costs are yours.

Official account & key contributors on X

• Andrej Karpathy — project author (@karpathy)

Table of Contents

Getting started

1. What it is and who it is for
2. Access and setup: your first session

Usage and configuration

3. A concrete task and how to check it
4. Costs: software, models, and execution
5. Configuration and extensions

Limits and references

6. Permissions, data, and limitations
7. Frequently asked questions
8. Official references

Use the contents to jump to each section. If an in-page link does not respond, open it in a new tab.

[ikaijua]

What it is and who it is for

In one sentence: A small research environment where an existing agent repeatedly changes training code and compares experiments.

Who it suits: People familiar with model training who have suitable GPU resources and want automated experiments.

What you can do:

• Propose and execute a training-code change.
• Compare validation metrics under a fixed time budget.
• Keep improvements and discard failed experiments.

Where to start: Clone the repository. Upstream requires an NVIDIA GPU, Python 3.10+, and uv; Mac forks have separate support.

Cost: MIT code; GPU, storage, and the external agent’s inference costs are yours.

[ikaijua]

Access and setup: your first session

Clone the repository. Upstream requires an NVIDIA GPU, Python 3.10+, and uv; Mac forks have separate support.

git clone https://github.com/karpathy/autoresearch.git
cd autoresearch
uv sync
uv run prepare.py
uv run train.py

Complete one manual preparation and training run before asking an existing agent to read program.md.

[ikaijua]

A concrete task and how to check it

Keep data and evaluation fixed and allow only one class of changes in train.py.

Example:

Read program.md and record the baseline. Compare two learning-rate settings, retaining configuration, logs, val_bpb, and conclusions. Do not change evaluation data or metrics.

Lower validation bits per byte (val_bpb) is better; it is not a general intelligence score.

[ikaijua]

Costs: software, models, and execution

The five-minute budget covers the training segment, not every startup, compilation, evaluation, or agent call. It does not guarantee a fixed number of experiments per hour.

Cloud instances may cost money while idle; agent subscriptions or APIs are separate. Set an experiment count and stopping rule.

[ikaijua]

Configuration and extensions

prepare.py supplies data, constants, and evaluation; train.py is the agent’s edit target; program.md defines the human-written research rules.

Keep evaluation fixed. The time budget supports comparisons on the same hardware; cross-hardware results are not automatically comparable.

[ikaijua]

Permissions, data, and limitations

This is not a hosted chatbot and does not provide a GPU. Run the external agent and training code in a dedicated environment.

Expect out-of-memory failures, dependency issues, no improvements, and possible overfitting to the validation setup. Keep failed runs and the baseline rather than only the best result.

[ikaijua]

Frequently asked questions

Does upstream run directly on a Mac? It targets NVIDIA; check listed community ports separately.

Can it research anything? The supplied task is small-model training; other domains need new code and evaluation.

Must the agent be unrestricted? No. Use a dedicated environment and explicit experiment boundaries.

[ikaijua]

Official references

• Repository and platform support
• program.md
• Training code
• MIT license