v0.3.0: Run campaigns locally and a CLI & chat interface

Faberon brain v0.3.0, on PyPI as faberon. Speaks API contract v0. Ships together with Faberon Chat v0.1.0.

Summary

Faberon v0.3.0 introduces a local executor that can run the experiments of a campaign in a local subprocess instead of submitting jobs to Slurm. It also introduces two new user interfaces: a CLI built with Typer, and a chat interface that is a Pi extension. Both now allow you to easily query the list of existing campaigns and their status, check the event log, inject a new idea mid-run and watch how your metric (hopefully) improves with each step.

Verified end to end against Karpathy's autoresearch. Workstation walkthrough: discussion #61.

What's included

• Local executor behind the existing submit/status/cancel interface, selected with an env switch
• Faberon Chat: a Pi extension with a tool and slash command per v0 operation
• faberon CLI (Typer): serve, create, list, show, events, cancel, chat
• Idea injection: POST /v0/campaigns/{id}/ideas lands in the ledger as idea.injected; the proposer weighs it at the next decision boundary
• Campaign creation from the CLI: submit a hand-written plan without curl
• The target file is now configurable in the plan, no longer hardcoded to train.py
• Security improvements (see also the v0.4.0 security follow-up)

Known limitations

• Ideas are advisory: the proposer reads them as context, nothing forces execution
• An idea injected mid-experiment does not interrupt the in-flight job
• Chat surfaces API errors as-is: no retry, no offline queue
• The chat's contract types are hand-maintained while v0 is still moving; they will be generated from OpenAPI later
• The local executor has no isolation or queueing: one subprocess at a time, no cgroup or container
• No database migration: upgrading to v0.3.0 requires recreating the database, existing ledger data is discarded

Pull requests

• #46 Update roadmap for the coming versions
• #48 Add two new event types: idea.injected and experiment.proposing
• #49 Record an experiment.designing event in the ledger
• #50 Allow a human to inject research/experiment ideas
• #51 Use MIN_PROSE_LENGTH as minimum length for free-text fields
• #54 Move command to ResearchPlan and make it str
• #53 Typer app for command-line interface
• #55 Add pi console for steering campaigns through a chat interface
• #56 Update readme, add separate installation and a troubleshooting guide
• #57 Small fixes in the documentation
• #58 Fix overview schema in readme
• #59 Extend the chat console to include many more tools
• #60 Add a local executor that runs campaigns on the workstation instead of Slurm
• #62 Design document for v0.3.1
• #63 Allow changing the single Python file that can be edited
• #64 Rename 'console' to Faberon 'chat'
• #65 Rename 'control-package' to Faberon 'brain'
• #66 Update readme and design docs
• #67 Security improvements and a targeted security release v0.4.0
• #68 Fix the formatting of non-verbose proposed events
• #69 Add a "faberon chat" command
• #72 Release v0.3.0
• #73 Add PyPI metadata and npm configuration

Try it

See the README for setup (workstation or login node), then the workstation runbook for a full local campaign.

v0.2.2: Fix threads syncing

Summary

Faberon v0.2.2 is a bugfix patch. PR #43 fixes the main bug by ensuring there's a dedicated worker loop for all agents to run on.

Pull requests

• #43 Have a dedicated worker loop for all agents to run on
• #44 Ensure that the proposer LLM provides a justification and a summary/title
• #45 Version bump to 0.2.2

v0.2.1: Robustness patch for the campaign loop

Summary

Faberon v0.2.1 is a robustness patch. The first v0.2.0 campaigns run on the cluster exposed some rough edges:

• two campaigns could corrupt one repo
• a crashed or hung campaign was indistinguishable from a slow one
• a proposer failure could kill a campaign
• the control plane was hard to stop and potentially unsafe to restart

This release fixes those by adding more robustness around error handling.

What's included

• One active campaign per repo: POST /v0/campaigns rejects a second campaign on a repo that already has one
• Informative commit messages: exp 3: extend LR warmdown 0.5 -> 0.7
• Proposer call timeout (FABERON_PROPOSER_TIMEOUT), so a hung model cannot freeze a campaign without being to cancel or restart it
• A proposer failure is not fatal: a no-op or invalid edit is caught, recorded, and retried or skipped
• A fixed ceiling of consecutive proposer failures ends the campaign with the error stop reason
• Unhandled loop exceptions record a terminal campaign.ended event (error stop reason); schema gains the error value and a campaign.crashed event
• Campaign status (active / ended / died) on GET /v0/campaigns and GET /v0/campaigns/{id}
• SSE streams close on shutdown, so the control plane stops without SIGKILL
• FABERON_NO_RECOVER start mode and POST /v0/campaigns/{id}/resume for safe restart and per-campaign recovery

Stand-ins (intentional)

• In-place resume of a crashed campaign is not supported; recovery is re-seeding from the ledger baseline
• Crash-cause diagnosis is manual: the status field says a campaign died, but finding why still means reading the DBOS error

⚠️ Caveat

There is no automated database schema migration: upgrading to this version requires recreating the database (scripts/db/drop-faberon-db.sh, then scripts/db/create-faberon-db.sh); existing campaign and ledger data is discarded

Pull requests

• #28 Add sha and index fields to ExperimentSetup
• #31 Ensure that the faberon_test DB exists in CI
• #32 Design patch release 0.2.1 focusing on robustness and UX
• #33 Ensure that one target repo can only have one running campaign
• #34 Organize bash scripts
• #35 Write an informative summary as git commit message
• #36 Make the calls to the LLM proposer more robust
• #38 Graceful termination of problematic campaigns
• #39 Derive campaign status from the ledger and DBOS
• #40 Ensure graceful shutdown of the faberon app when there's a live SSE stream
• #41 Avoid a potential crash loop by setting FABERON_NO_RECOVER and resuming manually
• #42 Version bump to 0.2.1

Try it

See the README for setup and the new "Safe restart and recovery" runbook.

v0.2.0: Autonomous campaign loop on Slurm

Summary

Faberon v0.2.0 runs an autonomous research campaign on a Slurm cluster. You hand-POST a plan (goal, metric, budget, caps) and a command. The control plane then loops: an LLM proposer edits train.py and commits, the job runs on Slurm, the metric is judged against the best so far, and the commit is kept or discarded. The loop repeats until the experiment cap or the GPU-hour budget fires, or you cancel.

Every step lands in the Postgres ledger, streamed live or read back as JSONL. Kill the control plane mid-campaign and restart it: the in-flight job is not resubmitted, the poll resumes, and the campaign finishes exactly once.

Verified end-to-end with Karpathy's autoresearch (5-minute train.py, val_bpb metric). Cluster walkthrough: discussion #23.

What's included

• New event types: experiment.proposed and campaign.ended
• run_campaign DBOS workflow: propose → run → judge → keep/discard → repeat
• LLM access via FABERON_MODEL: OpenRouter or a local OpenAI-compatible server
• Pydantic AI proposer with typed tools, which validates & writes a full-file replacement
• Git lineage: one commit per experiment. Keep advances the head; discard resets to the parent
• Hard stop conditions: manual cancel, GPU-hour budget, max_experiments
• Per-job walltime from the plan, clamped by FABERON_SLURM_MAX_TIME for additional security
• Campaign-scoped API
• Per-job Slurm output files, with a {job_id} placeholder in the metric command
• Dedicated faberon_test database for the test suite; opt-in Slurm and LLM integration tests

Stand-ins (intentional)

• Judgment is (still) a fixed rule
• Failed or cancelled jobs and unparseable metrics are discards that consume an experiment slot & contribute to the total time budget.
• Free-text stop conditions are recorded and visible, not enforced.
• One experiment in flight. No worktrees, no queue, no multi-file edits, no console (yet!).

Pull requests

• #14 Design document for v0.2.0
• #15 Set walltime on sbatch
• #16 Extend schema with stop reasons and max experiments
• #17 Git tree functionality: keep, commit, discard
• #18 Add CampaignRunner and more guard rails
• #19 Pydantic AI proposer for single-file experiments
• #21 Final refinements for v0.2.0

Try it

See the README for setup, then the cluster runbook for the full login-node walkthrough, and the kill-mid-campaign and cancel drills in #24.

v0.1.0: First durable experiment path on Slurm

Summary

Faberon v0.1: one thin path through every technical tier, proven durable on a real Slurm login node.

You hand-POST a plan and a command. The control plane validates it, submits a Slurm job, polls to completion, parses a metric, applies a keep/discard judgment, and records everything in a Postgres ledger streamed over SSE. Kill the control plane mid-run, restart it, and the job is not resubmitted: the poll resumes and the result is judged and logged exactly once.

Verified end-to-end with Karpathy's autoresearch (5-minute train.py, val_bpb metric). Cluster walkthrough: discussion #12.

What's included

• Monorepo scaffold (uv, packages/control-plane)
• Quarantined schema: event record + research plan models
• Executor interface (submit / status / cancel) with a fake shim for tests and a real Slurm adapter
• Postgres event ledger (append-only, campaign-scoped queries, SSE tail)
• DBOS durable workflow: submit → poll → parse → judge → record (one concurrent job)
• FastAPI surface: POST /v0/campaigns, GET /v0/campaigns/{id}, GET /v0/events, GET /healthz
• Bearer token auth (mandatory for the Slurm entrypoint)
• faberon CLI (uv tool install packages/control-plane)
• Postgres setup scripts for dev machines and no-sudo login nodes

Stand-ins (intentional)

• Judgment is a fixed rule: keep if the metric is below the baseline (lower is better). No confidence measure, no constraints.
• Poll interval is a fixed value from the caller. No adaptive sacct polling.
• One concurrent job. No agent loop, no train.py editing, no console.

Pull requests

• #1 Repo scaffold, design docs, contributing guidelines
• #2 Add a minimal Pydantic schema for research plans and event records
• #3 Add a Protocol for the job executor
• #4 Add tooling for formatting and linting
• #5 Implement SlurmExecutor
• #6 Event ledger backed by Postgres
• #8 Add output option for sbatch submissions
• #10 Runtime object for a durable experiment workflow
• #11 Add FastAPI interface
• #13 Final cleanup for v0.1.0 runbook

Try it

See the README for setup, then the cluster runbook for the full login-node walkthrough including the kill-mid-run exit test.