v0.3: autoresearch runbook on a local workstation

[svlandeg]

This runbook walks you through running Faberon v0.3 against Karpathy's autoresearch using a local workstation. It will show you how to use the new CLI and chat interface.

What you need

• uv installed in your home directory.
• An OpenRouter API key or any OpenAI-compatible endpoint.
• git with your identity (email, name) configured: Faberon creates one commit per experiment.

1. Checkouts

Clone both repos:

mkdir -p ~/git
cd ~/git
git clone https://github.com/sevren-ai/faberon
git clone https://github.com/karpathy/autoresearch

The commands below assume both clones live in ~/git. Adjust the cd paths if you put them elsewhere.

2. Install prerequisites

Set the environment, install Postgres, Node and Pi as explained in the workstation guide.

Note that the runbook as shown here has used openrouter:moonshotai/kimi-k3 as the brain's LLM.

3. Install Faberon

Install the brain and the chat interface:

cd ~/git/faberon
git checkout v0.3.0 -b v0.3.0

uv tool install packages/brain
pi install ./packages/chat

If you had a previous version of Faberon installed, and you don't have any data from previous campaigns that you want to persist, it's best to recreate the DB from scratch:

bash scripts/db/drop-faberon-db.sh
bash scripts/db/create-faberon-db.sh

4. Prepare autoresearch

cd ~/git/autoresearch
git checkout -b my_experiment
uv run prepare.py

The scratch branch keeps Faberon's commits off master. prepare.py downloads the training data.

5. Measure the baseline

The plan needs a baseline metric to judge the first experiment against. Run the stock train.py once and check the output

cd ~/git/autoresearch
uv run train.py

Wait for it (about 5 to 10 minutes), then read the metric from the output, something like:

val_bpb:          1.253713

⚠️ If the run crashes due to OOM, edit the code to change DEVICE_BATCH_SIZE in train.py to something smaller, e.g. 32.

6. Define the research plan

Write the research plan, replacing <baseline> with your measured val_bpb from the previous step:

cd ~/git/autoresearch
mkdir .faberon
cat > .faberon/plan.json <<EOF
{
  "goal": "Beat the measured val_bpb baseline on autoresearch.",
  "command": "uv run train.py",
  "metric": "val_bpb",
  "target_file": "train.py",
  "baseline": <baseline>,
  "budget_gpu_hours": 2.0,
  "max_experiments": 6,
  "max_concurrency": 1,
  "walltime": 10,
  "stop_conditions": ["Stop after 6 experiments or 2 GPU-hours, whichever comes first."]
}
EOF

7. Start Faberon

cd ~/git/autoresearch
faberon serve

8. Create a campaign and track it

With Faberon now running, start a campaign using the plan we just drafted:

cd ~/git/autoresearch
faberon create .faberon/plan.json 

This will print the campaign ID.

Then track the campaign through the CLI:

faberon --help
faberon list
faberon show <campaign_id>
faberon events <campaign_id> -v

Or use the chat interface to check it:

faberon chat

(example output, cf first comment below 👇 )

[svlandeg]

Example campaign, querying its status through the chat interface and steering experiments by injecting ideas:

When asking for more information on that campaign:

And now we can ask about all kinds of experiments we'd like to do:

Let's say something like "yes, inject it", then check our events again!

Then let's wait a bit and check on the events again a little later, and it turns out our idea was indeed tested but didn't make the cut:

and we get some additional context & information: