v0.2: autoresearch runbook on Slurm

[svlandeg]

⚠️ Note that this runbook is now outdated and is only kept as reference.
Use the runbook for the current version instead. ⚠️

---

This runbook walks you through running Faberon v0.2 against Karpathy's autoresearch on a real Slurm cluster. By the end you will have run one autonomous campaign of 6 experiments, with an LLM-in-the-loop deciding which experiments to try.

Budget about one hour, mostly waiting on jobs. Cost: roughly 0.5 GPU-hour and a handful of model calls.

What happens in a campaign: you (manually) POST a research plan to the control plane. The control plane loops: the proposer (an LLM agent) reads train.py and the full campaign history, writes a replacement train.py, commits it, submits it as a Slurm job, parses the metric, and keeps or discards the commit. Every step lands in a Postgres ledger. The loop stops when it reaches the maximum number of experiments or the GPU-hour budget, or when you cancel it.

What you need

• A Slurm login node: sbatch, squeue, sacct on PATH, a Slurm account to bill to, and your home directory on the shared cluster filesystem.
• uv installed in your home directory.
• An OpenRouter API key. Any OpenAI-compatible endpoint works too; the README lists the env vars.
• git with your identity (email, name) configured: Faberon creates one commit per experiment.

No sudo is needed anywhere.

1. Checkouts (once)

Clone both repos onto the shared filesystem, so compute nodes see the same tree as the login node:

mkdir -p ~/git
cd ~/git
git clone https://github.com/sevren-ai/faberon
git clone https://github.com/karpathy/autoresearch

The commands below assume both clones live in ~/git. Adjust the cd paths if you put them elsewhere.

2. Postgres (once per machine)

The control plane stores its ledger in Postgres. On a login node, install a personal server:

cd ~/git/faberon
git checkout v0.2.2 -b v0.2.2
bash scripts/install-postgres-no-sudo.sh
bash scripts/start-postgres.sh
bash scripts/create-faberon-db.sh

The installer records the port in ~/.config/faberon/postgres.env (as PGPORT, and inside FABERON_DATABASE_URL). You need the URL for the env file below. If you installed Postgres earlier, skip the install command; the file is still there.

3. Environment (once)

Faberon reads all configuration from environment variables. Put them in one file, readable only by you:

mkdir -p ~/.config/faberon
cat > ~/.config/faberon/model.env <<'EOF'
export FABERON_MODEL=openrouter:<provider>/<model>
export OPENROUTER_API_KEY=<key>
export FABERON_SLURM_ACCOUNT=<account>
export FABERON_DATABASE_URL=postgres://localhost:<port>/faberon
export FABERON_API_TOKEN=<random_token>
EOF
chmod 600 ~/.config/faberon/model.env

Notes:

• <port> is the one from step 2: grep FABERON_DATABASE_URL ~/.config/faberon/postgres.env. The URL appears in both env files on purpose. Only the Postgres scripts source postgres.env; your shell sources model.env. If you ever reinstall Postgres on a new port, update the URL here too.
• Generate the token with python3 -c "import secrets; print(secrets.token_urlsafe(32))". The token guards the API: on a shared login node, every user can reach localhost.
• The key never enters the repo or the ledger. The model provider reads it from the environment at startup.

Load it in every shell:

echo '[ -f ~/.config/faberon/model.env ] && . ~/.config/faberon/model.env' >> ~/.bashrc
source ~/.bashrc

4. Pre-flight checks

Two opt-in test suites verify the Slurm path and the model path end to end:

cd ~/git/faberon
bash scripts/slurm-test.sh
bash scripts/llm-test.sh

The tests run against a separate faberon_test database (created by create-faberon-db.sh), so they never touch the real ledger.

The first submits real (tiny) jobs. The second makes one real model call and checks the proposer reads its context and returns valid Python.

There should be no skipped or failing tests. Fix whatever fails here first, before spending GPU time on the baseline.

5. Install the faberon command (once)

cd ~/git/faberon
uv tool install packages/control-plane

Re-run this after pulling a new Faberon version.

6. Prepare autoresearch (once)

cd ~/git/autoresearch
git checkout -b my_experiment
uv run prepare.py

The scratch branch keeps Faberon's commits off master. prepare.py downloads the training data.

7. Measure the baseline (once)

The plan needs a baseline metric to judge the first experiment against. Run the stock train.py once, as a manual Slurm job. Faberon keeps its scratch files (job output, the campaign file) in a .faberon/ directory in the checkout, untracked by git:

cd ~/git/autoresearch
mkdir -p .faberon
sbatch --account="$FABERON_SLURM_ACCOUNT" --gres=gpu:1 --time=10 --output="$PWD/.faberon/baseline.out" --wrap "uv run train.py"

This run should prove the training stack works on a GPU node before autonomy starts.
Wait for it (about 5 to 10 minutes; watch squeue -u $USER), then read the metric:

grep val_bpb .faberon/baseline.out

Note the value.
If there is no val_bpb line and the job isn't running anymore, it hit the walltime: raise --time (expressed in minutes) and rerun.

When the run finished successfully, inspect the runtime in seconds:

grep total_seconds .faberon/baseline.out

If total_seconds is close to the time limit, raise walltime (in minutes) in the plan below; the proposer's edits can make later runs slower than stock.

8. Define the campaign

Write the plan, replacing <baseline> with your measured val_bpb from the previous step:

cat > $HOME/git/autoresearch/.faberon/campaign.json <<EOF
{
  "campaign_id": "$(python3 -c 'import uuid; print(uuid.uuid4())')",
  "plan": {
    "goal": "Beat the measured val_bpb baseline on autoresearch.",
    "metric_name": "val_bpb",
    "metric_command": "cat $HOME/git/autoresearch/.faberon/faberon-{job_id}.out",
    "baseline": <baseline>,
    "budget_gpu_hours": 2.0,
    "max_experiments": 6,
    "max_concurrency": 1,
    "walltime": 10,
    "stop_conditions": ["Stop after 6 experiments or 2 GPU-hours, whichever comes first."]
  },
  "command": ["uv", "run", "train.py"],
  "poll_interval_seconds": 15,
  "repo_path":$HOME/git/autoresearch"
}
EOF

9. Start the control plane

One more setting first, in the same shell that will run faberon:

export FABERON_SLURM_OUTPUT="$HOME/git/autoresearch/.faberon/faberon-%j.out"

FABERON_SLURM_OUTPUT sets each job's stdout file. Slurm expands %j to the job ID, so every experiment gets its own output file.

Now, let's start the control plane. The target repo is the working directory, so start it from the autoresearch checkout in a tmux pane so it survives your connection:

cd ~/git/autoresearch
faberon

Check liveness from another pane:

curl http://127.0.0.1:8000/healthz

it should print

{"status":"ok"}

Also check the running server matches this runbook's version:

curl -H "Authorization: Bearer $FABERON_API_TOKEN" http://127.0.0.1:8000/openapi.json | python3 -c 'import json,sys; print(json.load(sys.stdin)["info"]["version"])'

It should print 0.2.2. Anything older means the faberon command is a stale install: cd ~/git/faberon && git pull && uv tool install --reinstall packages/control-plane, then restart it.

10. Submit the campaign

Now, submit the manually drafted campaign plan to the Faberon control plane:

cd ~/git/autoresearch
curl -X POST http://127.0.0.1:8000/v0/campaigns -H "Authorization: Bearer $FABERON_API_TOKEN" -H "Content-Type: application/json" -d @$HOME/git/autoresearch/.faberon/campaign.json

Expect a JSON response with the campaign ID and workflow ID.

11. Inspect the campaign outputs

You can inspect the current progress of the campaign by live-streaming the events in the ledger:

curl -N -H "Authorization: Bearer $FABERON_API_TOKEN" http://127.0.0.1:8000/v0/campaigns/{id}/events

(replace {id} with the campaign ID that is also recorded in .faberon/campaign.json)

First, you'll see a campaign.created event. Then, the control plane will start conducting experiments.

Each experiment produces three events: experiment.proposed (the commit SHA and the proposer's rationale), experiment.completed (job ID, state, exit code, elapsed seconds), and experiment.judged (metric, keep or discard, and why). Meanwhile:

• squeue -u $USER shows the current job, named faberon:<campaign_id>:<n>.
• .faberon/faberon-<job_id>.out collects each job's stdout. The {job_id} in the metric command is replaced with the actual job ID before the command runs, so each experiment is judged on its own output.

The stream never terminates on its own, which makes it awkward to search. To read the ledger instead of watching it, fetch it as JSON Lines:

curl -N -H "Authorization: Bearer $FABERON_API_TOKEN" http://127.0.0.1:8000/v0/campaigns/{id}/events.jsonl > .faberon/ledger.jsonl

(again, replace {id} with the campaign ID that is also recorded in .faberon/campaign.json)

The file holds one JSON event per line, so grep experiment.judged .faberon/ledger.jsonl and friends work.

Troubleshooting

• curl: 401 unauthorized: wrong or missing bearer token.
• 422 JSON decode error on submit: curl needs -d @<file>; without the @ it sends the path itself as the body. Also check that <baseline> was replaced in campaign.json.
• RuntimeError: FABERON_MODEL is not set at startup: the env file is not sourced in this shell. source ~/.config/faberon/model.env.
• UserError about OPENROUTER_API_KEY at startup: the key is missing or does not match the FABERON_MODEL prefix.
• Postgres connection refused: bash scripts/start-postgres.sh.
• sbatch: command not found: you are not on the login node, or your shell lost the Slurm PATH. Try a login shell (exec bash -l).
• Jobs fail instantly: read the job's .faberon/faberon-<job_id>.out in the checkout. Usually uv is not on PATH in the job; sbatch inherits the submitting shell's environment, so start faberon from a shell that sourced ~/.bashrc.

Q&A

Questions and run reports welcome below.

Crash & cancel drills: #24

[svlandeg]

Example output (bolding by me):

Cherrypicking results:

{"seq":5,"ts":"2026-09-21T11:00:38.211195+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.proposed","justification":"Single focused change: halve TOTAL_BATCH_SIZE from 2**19 to 2**18 tokens (grad accum 2→1). At this short wall-time (~5 min), the model is heavily optimizer-step-limited rather than token-limited: halving batch size doubles the number of Muon/AdamW updates for the same compute, which consistently improves final loss at fixed tokens-processed in this regime. The previous schedule-tuning attempt (longer warmdown) failed, so we return to the baseline WARMDOWN_RATIO=0.5 and change only the batch size to keep the delta attributable. MATRIX_LR is kept at 0.04 since Muon's update norm is approximately batch-size-invariant.","payload":{"sha":"5dadac90d339df824dcc3a5f5f18f68ca19e9584","index":2,"parent_sha":"228791fb499afffb54b46200aca536f79142f117"}}

{"seq":6,"ts":"2026-09-21T11:06:25.890566+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.completed","justification":"job 508943 completed exit=0","payload":{"state":"completed","job_id":"508943","exit_code":0,"elapsed_seconds":336.0}}

{"seq":7,"ts":"2026-09-21T11:06:25.896374+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.judged","justification":"metric=0.986541 baseline=0.995727 judgment=keep","payload":{"job_id":"508943","reason":"metric 0.986541 beats best 0.995727","baseline":0.995727,"judgment":"keep","metric_value":0.986541}}

Full details:

Details

{"seq":1,"ts":"2026-09-21T10:45:47.837872+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"human","type":"campaign.created","justification":"plan accepted","payload":{"goal":"Beat the measured val_bpb baseline on autoresearch.","baseline":0.995727,"walltime":15,"metric_name":"val_bpb","metric_command":"cat /dcai/users/lansof/git/autoresearch/.faberon/faberon-{job_id}.out","max_concurrency":1,"max_experiments":6,"stop_conditions":["Stop after 6 experiments or 2 GPU-hours, whichever comes first."],"budget_gpu_hours":2.0}}

{"seq":2,"ts":"2026-09-21T10:49:59.120707+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.proposed","justification":"Single focused change: extend the LR warmdown phase from 50% to 70% of the time budget. Nanochat-style trapezoid schedules consistently show that a longer linear decay to ~0 LR lowers final val_bpb at fixed compute, since most of the loss improvement comes from the annealing tail. All other hyperparameters and the architecture are left unchanged, so any val_bpb delta is cleanly attributable to the schedule change.","payload":{"sha":"f17ad0c744a6cde7652b713ea0d2d7817b9fa437","index":1,"parent_sha":"228791fb499afffb54b46200aca536f79142f117"}}

{"seq":3,"ts":"2026-09-21T10:55:46.889047+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.completed","justification":"job 508924 completed exit=0","payload":{"state":"completed","job_id":"508924","exit_code":0,"elapsed_seconds":339.0}}

{"seq":4,"ts":"2026-09-21T10:55:46.903703+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.judged","justification":"metric=0.996349 baseline=0.995727 judgment=discard","payload":{"job_id":"508924","reason":"metric 0.996349 does not beat best 0.995727","baseline":0.995727,"judgment":"discard","metric_value":0.996349}}

{"seq":5,"ts":"2026-09-21T11:00:38.211195+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.proposed","justification":"Single focused change: halve TOTAL_BATCH_SIZE from 2**19 to 2**18 tokens (grad accum 2→1). At this short wall-time (~5 min), the model is heavily optimizer-step-limited rather than token-limited: halving batch size doubles the number of Muon/AdamW updates for the same compute, which consistently improves final loss at fixed tokens-processed in this regime. The previous schedule-tuning attempt (longer warmdown) failed, so we return to the baseline WARMDOWN_RATIO=0.5 and change only the batch size to keep the delta attributable. MATRIX_LR is kept at 0.04 since Muon's update norm is approximately batch-size-invariant.","payload":{"sha":"5dadac90d339df824dcc3a5f5f18f68ca19e9584","index":2,"parent_sha":"228791fb499afffb54b46200aca536f79142f117"}}

{"seq":6,"ts":"2026-09-21T11:06:25.890566+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.completed","justification":"job 508943 completed exit=0","payload":{"state":"completed","job_id":"508943","exit_code":0,"elapsed_seconds":336.0}}

{"seq":7,"ts":"2026-09-21T11:06:25.896374+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.judged","justification":"metric=0.986541 baseline=0.995727 judgment=keep","payload":{"job_id":"508943","reason":"metric 0.986541 beats best 0.995727","baseline":0.995727,"judgment":"keep","metric_value":0.986541}}

{"seq":8,"ts":"2026-09-21T11:10:01.106640+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.proposed","justification":"Single focused change: halve TOTAL_BATCH_SIZE again, 2**18 → 2**17, and set DEVICE_BATCH_SIZE=64 so grad-accum stays at 1. Experiment 2 showed that halving the batch size (doubling optimizer steps) gave the largest gain so far (0.9957
0.9865), confirming the ~5-minute run is optimizer-step-limited. Doubling the update count again is the natural next step; Muon's update norm is roughly batch-size-invariant so LRs are kept unchanged. All other hyperparameters and the architecture are untouched, keeping the delta cleanly attributable.","payload":{"sha":"3cbccbe5d3536ce291bb7f2b4d800c793c5993e8","index":3,"parent_sha":"5dadac90d339df824dcc3a5f5f18f68ca19e9584"}}

{"seq":9,"ts":"2026-09-21T11:15:51.823494+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.completed","justification":"job 508951 completed exit=0","payload":{"state":"completed","job_id":"508951","exit_code":0,"elapsed_seconds":349.0}}

{"seq":10,"ts":"2026-09-21T11:15:51.836671+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.judged","justification":"metric=0.990748 baseline=0.986541 judgment=discard","payload":{"job_id":"508951","reason":"metric 0.990748 does not beat best 0.986541","baseline":0.986541,"judgment":"discard","metric_value":0.990748}}

{"seq":11,"ts":"2026-09-21T11:19:40.751514+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.proposed","justification":"Single focused change: reduce EMBEDDING_LR from 0.6 to 0.4, keeping the winning config (TOTAL_BATCH_SIZE=2**18, WARMDOWN_RATIO=0.5) from experiment 2 (val_bpb 0.986541). The 0.6 embedding LR was tuned for the 2**19 batch size; after halving the batch (doubling optimizer steps), the effective embedding update magnitude is roughly doubled, so scaling the LR down proportionally (~×0.67) should reduce embedding oscillation and improve the annealed final loss. Everything else is unchanged so the delta is attributable. Experiment 3 showed 2**17 is too small (too noisy), so batch size stays at 2**18.","payload":{"sha":"3a0af4fe6e7b4ee10c5b84afb230dfeb77002458","index":4,"parent_sha":"5dadac90d339df824dcc3a5f5f18f68ca19e9584"}}

{"seq":12,"ts":"2026-09-21T11:25:31.707478+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.completed","justification":"job 508956 completed exit=0","payload":{"state":"completed","job_id":"508956","exit_code":0,"elapsed_seconds":336.0}}

{"seq":13,"ts":"2026-09-21T11:25:31.717586+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.judged","justification":"metric=0.987088 baseline=0.986541 judgment=discard","payload":{"job_id":"508956","reason":"metric 0.987088 does not beat best 0.986541","baseline":0.986541,"judgment":"discard","metric_value":0.987088}}

{"seq":14,"ts":"2026-09-21T11:34:43.915336+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.proposed","justification":"Single focused change: raise MATRIX_LR from 0.04 to 0.05 on top of the current best config (TOTAL_BATCH_SIZE=2**18, EMBEDDING_LR=0.6, WARMDOWN_RATIO=0.5 → val_bpb 0.986541). The run is optimizer-step-limited (halving batch to 2**18 gave the biggest gain, 0.9957→0.9865), so more aggressive Muon updates should help; 0.04 was tuned for the 2**19 batch. Embedding LR tuning (0.4) and further batch halving (2**17) both regressed, so those axes are exhausted; schedule changes (warmdown 0.7) also failed at 2**19. This touches only one scalar, keeping the delta attributable.","payload":{"sha":"f44cd1d95d1197a7d23b57c2c59c23fa7b4eada2","index":5,"parent_sha":"5dadac90d339df824dcc3a5f5f18f68ca19e9584"}}

{"seq":15,"ts":"2026-09-21T11:40:32.976495+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.completed","justification":"job 508986 completed exit=0","payload":{"state":"completed","job_id":"508986","exit_code":0,"elapsed_seconds":334.0}}

{"seq":16,"ts":"2026-09-21T11:40:32.986609+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.judged","justification":"metric=0.989252 baseline=0.986541 judgment=discard","payload":{"job_id":"508986","reason":"metric 0.989252 does not beat best 0.986541","baseline":0.986541,"judgment":"discard","metric_value":0.989252}}

{"seq":17,"ts":"2026-09-21T11:44:58.183427+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.proposed","justification":"Single focused change on top of the current best config (TOTAL_BATCH_SIZE=2**18, EMBEDDING_LR=0.6, MATRIX_LR=0.04, WARMDOWN_RATIO=0.5 → val_bpb 0.986541): raise UNEMBEDDING_LR from 0.004 to 0.006. This is the only LR axis not yet probed. In the step-limited short-horizon regime (halving the batch was the biggest win so far, while EMBEDDING_LR and MATRIX_LR increases regressed), the lm_head's tiny Adam LR is the least-tuned default; a modest 1.5x increase should let the unembedding adapt faster to the embedding/feature drift within the annealed window, potentially lowering final val_bpb. All other hyperparameters and the architecture are untouched, so the delta is cleanly attributable.","payload":{"sha":"7739182ea6ae6f847e850228775ea02256b045f6","index":6,"parent_sha":"5dadac90d339df824dcc3a5f5f18f68ca19e9584"}}

{"seq":18,"ts":"2026-09-21T11:50:37.666200+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.completed","justification":"job 508996 completed exit=0","payload":{"state":"completed","job_id":"508996","exit_code":0,"elapsed_seconds":334.0}}

{"seq":19,"ts":"2026-09-21T11:50:37.715444+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"experiment.judged","justification":"metric=0.987147 baseline=0.986541 judgment=discard","payload":{"job_id":"508996","reason":"metric 0.987147 does not beat best 0.986541","baseline":0.986541,"judgment":"discard","metric_value":0.987147}}

{"seq":20,"ts":"2026-09-21T11:50:40.307545+02:00","campaign_id":"076f254e-9799-4a5c-a04f-2d3a1b60c525","actor":"agent","type":"campaign.ended","justification":"campaign ended: max_experiments","payload":{"stop_reason":"max_experiments","experiments_done":6,"gpu_hours_burned":0.5633333333333334}}