v0.1: cluster runbook

[svlandeg]

⚠️ Note that this runbook is now outdated and is only kept as reference.
Use the runbook for the current version instead. ⚠️

---

This runbook walks you through an artificial "campaign" on a Slurm login node using Karpathy's autoresearch: start the control plane, POST a plan, watch the ledger stream, confirm the 5-minute training job ran and was judged. Ends with the kill-mid-run exit test, where the 5-minute budget gives plenty of room to interrupt and restart.

This runbook uses two shells (two tmux panes, or two terminals). Shell 1 runs the control plane. Shell 2 prepares the workload and submits the campaign.

This runbook still contains a lot of manual steps that will be automated in the next Faberon versions, when LLMs will start driving the experiments. For now, it just checks that all the mechanical bits work correctly.

Prerequisites

Ensure that there is:

• sbatch, sacct, squeue, scancel on your PATH.
• A Slurm account you can bill jobs to.
• A single-GPU partition (autoresearch targets one NVIDIA GPU, tested on H100).

Shell 1: set up and start the control plane

One-time setup

Clone the Faberon repo and install Postgres:

git clone https://github.com/sevren-ai/faberon
cd faberon
git checkout v0.1.0 -b v0.1.0
bash scripts/install-postgres-no-sudo.sh

Start Postgres and create the Faberon DB:

bash scripts/start-postgres.sh
bash scripts/create-faberon-db.sh   # once, after install

Install the faberon command:

uv tool uninstall faberon
uv tool install packages/control-plane

Start the server (each session)

Source the Postgres env (sets FABERON_DATABASE_URL and puts the Postgres binaries on PATH), then set the Slurm account and API token:

source ~/.config/faberon/postgres.env
export FABERON_SLURM_ACCOUNT=<account>
export FABERON_API_TOKEN=$(python -c "import secrets; print(secrets.token_urlsafe(32))")
# export FABERON_SLURM_GPUS=1   # GPUs per job (default 1)
echo "API token: $FABERON_API_TOKEN"   # note this; shell 2 needs the same value
faberon

The server binds to 127.0.0.1:8000. Override with FABERON_HOST and FABERON_PORT if needed.

Shell 2: prepare the workload and submit a campaign

Open a new shell or tmux pane. This is where you clone autoresearch, build the campaign JSON, and submit it.

Prepare the autoresearch checkout (one-time)

Clone autoresearch onto the shared filesystem so the login node and compute nodes see the same tree:

git clone https://github.com/karpathy/autoresearch
cd autoresearch
uv sync
uv run prepare.py        # one-time: downloads data, trains tokenizer (~2 min)
mkdir .faberon

Check Faberon is listening

Check liveness in a quick curl (no token needed):

curl -s http://127.0.0.1:8000/healthz
# {"status":"ok"}

Submit a campaign

Set the API token to the same value shell 1 printed:

export FABERON_API_TOKEN=<the token shell 1 printed>

Create a file campaign.json with this content. Replace /path/to/autoresearch with your checkout path (twice) and optionally campaign_id with a fresh UUID (python -c "import uuid; print(uuid.uuid4())"):

{
  "campaign_id": "00000000-3333-6666-9999-000000000001",
  "plan": {
    "goal": "Beat the measured val_bpb baseline on autoresearch.",
    "metric_name": "val_bpb",
    "metric_command": "grep val_bpb /path/to/autoresearch/.faberon/run.log",
    "baseline": 2.0,
    "budget_gpu_hours": 1.0,
    "max_concurrency": 1,
    "stop_conditions": ["one run"]
  },
  "command": ["sh", "-c", "cd /path/to/autoresearch && set -o pipefail && uv run train.py 2>&1 | tee .faberon/run.log"],
  "poll_interval_seconds": 15.0
}

The baseline of 2.0 is a placeholder so the first run always keeps (any real val_bpb is below 2). For real campaigns, set baseline to a prior run's val_bpb you want to beat.

Submit it:

curl -s -X POST http://127.0.0.1:8000/v0/campaigns \
  -H "Authorization: Bearer $FABERON_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d @campaign.json

A 201 response returns campaign_id and workflow_id. Set CAMP_ID for the commands below:

export CAMP_ID=<the campaign_id from campaign.json>

Watch the ledger stream

curl -N -H "Authorization: Bearer $FABERON_API_TOKEN" \
  http://127.0.0.1:8000/v0/events?after=0

If you keep it open for 5-6 minutes, you'll see the following events getting streamed in:

• event: ledger with "type":"campaign.created"
• event: ledger with "type":"experiment.completed" (after the 5-minute run finishes)
• event: ledger with "type":"experiment.judged" and "payload":{"judgment":"keep"}

The job trained for 5 minutes, printed val_bpb: <value> (well below 99), so the judge keeps it.

Confirm the job ran on Slurm

The job name is faberon:<campaign_id>. Confirm it in Slurm:

sacct -X --name "faberon:$CAMP_ID" -P -o JobID,State,ExitCode,Elapsed

Expect one row, COMPLETED, exit 0:0, elapsed near 00:05:00.

Confirm the metric and judgment

The training output is in .faberon/run.log inside your autoresearch checkout. Check the metric line:

grep val_bpb run.log
# val_bpb:          1.234567

Confirm the judgment in the ledger:

curl -s -H "Authorization: Bearer $FABERON_API_TOKEN"  http://127.0.0.1:8000/v0/campaigns/$CAMP_ID | python -m json.tool

The response lists the campaign's events in order. The last one is experiment.judged with judgment: keep.

Exit test: kill mid-run, restart, resume

This is the v0.1.0 exit test. The 5-minute training budget gives a wide window to interrupt the control plane while the Slurm job is still running.

1. In shell 2, start a fresh campaign by changing the campaign_id in campaign.json and updating $CAMP_ID

2. POST the new campaign and watch squeue until the job is RUNNING:

   squeue -u $USER

3. While the job is RUNNING (anytime in the first ~5 minutes), kill the control plane in shell 1: Ctrl-C the faberon process.

4. Confirm the Slurm job is still running. The job is independent of the control plane.

5. In shell 1, restart the control plane:

   faberon

6. DBOS recovers the workflow on launch. It does not resubmit the job (submission is idempotent on the job name). It resumes polling sacct. When the job completes, it parses val_bpb from .faberon/run.log and records experiment.completed and experiment.judged exactly once.

7. In shell 2, confirm with the SSE stream or GET /v0/campaigns/$CAMP_ID: exactly one experiment.completed and one experiment.judged, no duplicates. The Slurm job ran exactly once (sacct shows one job, not two).