v0.6.14

• Add --start-point-idempotency-key (or BENCHER_START_POINT_IDEMPOTENCY_KEY) to bencher run, bencher report create, and bencher branch update, so parallel jobs that reset a Branch with the same key share one Head; in GitHub Actions, bencher run defaults the key to the workflow run attempt
• Fix concurrent runs on a Branch failing when they create the same Thresholds, such as parallel runs that clone Thresholds from a start point; the first run creates them and the rest use them
• Add GET /v0/server/health, which checks that the API server runs and reaches its database
• BREAKING CHANGE Remove GET /v0/projects. List an organization's projects with GET /v0/organizations/{organization}/projects, which bencher project list now calls with a required organization argument, and the MCP list_projects tool now takes an organization and asks an anonymous caller to sign in
• Add --image-cache-size to runner up (or BENCHER_IMAGE_CACHE_SIZE), the bytes of Image config and layer blobs a Bare Metal Runner keeps across Jobs in blobs under its --state-dir, so a Job downloads only the blobs the Runner does not already hold, while its manifest still comes from the registry. Between Jobs the least recently used blobs are removed until the rest fit, a Runner given a size exits at startup if it cannot make the store its own, and without one the Runner keeps none
• A Bare Metal Job whose image cannot be pulled or prepared now fails, even with --allow-failure, instead of completing with no results
• A Bare Metal build time now covers the image's command alone, timed where the command runs, and no longer includes the image pull, unpack, and boot; only an iteration whose command succeeds reports one
• Support Hyperfine 2 JSON output in the shell_hyperfine adapter (Thank you mrazauskas)
• Fix a very large page query parameter returning the wrong page
• BREAKING CHANGE Cap perf queries at 8 branches, testbeds, benchmarks, and measures each, and at 64 permutations and 64 lines; extra entries are ignored rather than refused
• Add a plus_limit server config option for the daily metrics limit of organizations with a Bencher Plus plan
• BREAKING CHANGE Cap a plot at 8 branches, testbeds, benchmarks, and measures each
• Cap active Thresholds at 64 per Branch, Testbed, and Measure, and skip the Thresholds a report cannot create (over a cap or over the daily creation limit) instead of failing the report
• BREAKING CHANGE The plus.litestream server config key is removed; use plus.disaster_recovery. A config that still uses plus.litestream is ignored and disaster recovery does not run at all: no restore at startup and no replication
• BREAKING CHANGE The sftp disaster recovery replica scheme is removed; file and s3 remain
• BREAKING CHANGE The on demand database backup is removed: POST /v0/server/backup, bencher server backup, and the database.data_store server config section
• BREAKING CHANGE The container registry storage config moves from plus.registry.data_store (service: aws_s3, an S3 access point ARN) to plus.registry.storage (scheme: file | s3, a bucket plus optional endpoint, region, and path), and any S3-compatible object store now works. Stored keys and the local directory layout are unchanged for a plain prefix, so existing data is picked up as is; a path that began or ended with a slash is now normalized, and one containing a character the store percent-encodes is now encoded, so see the config reference before upgrading. A config that still carries data_store is ignored and the registry falls back to local files
• BREAKING CHANGE A Bare Metal Runner that executes sandboxed Jobs must now run as root, where before it did not. Confining the Firecracker VMM requires privileges that enabling KVM does not grant: mknod to create the sandbox's /dev/kvm, chown to hand the guest images to the sandbox user, pivot_root, and setns to join an empty network namespace. A world-readable /dev/kvm is enough to use KVM unprivileged but not to build the sandbox around it. Start runner up as root, or as a service that starts it as root. If you cannot run as root, start the Runner with --danger-allow-no-sandbox and assign it only Specs with no Sandbox; be aware that this gives up the microVM itself and not just its confinement, so those Jobs execute directly on the host
• Confine the Firecracker VMM on Bare Metal Runners with the Firecracker jailer: the VMM behind each sandboxed Job now runs in a chroot as a dedicated unprivileged user, in an empty network namespace, and is placed in its cgroup before it starts, so a VMM escape no longer lands as root on the Runner host holding the Runner key and every prior Job's work. The user defaults to 61016 and is settable with --jail-uid/--jail-gid, neither of which may be 0
• Add --state-dir to the Runner (default /var/lib/bencher-runner), a persistent directory holding the chroot each sandboxed Job's VMM runs in. Before every sandboxed Job it is swept of anything a previous Runner left behind, since a Runner killed mid-Job cannot clean up after itself. A host runs one Runner at a time, whatever its state directory: a Runner started as root holds /run/bencher/runner.lock for as long as it runs, and a second one started as root exits at once with an error naming that lock; a Runner that is not root cannot open the lock and runs without it. When a Runner started as root comes up, and again before each sandboxed Job, it kills any process still running in a cgroup under /sys/fs/cgroup/bencher, such as the sandbox of a Runner killed mid-Job, whatever that Runner's state directory
• BREAKING CHANGE Sandboxed Jobs on Bare Metal Runners now need Linux 5.14 or later and a writable cgroup v2 hierarchy on every host, a one-core host included: each sandboxed Job runs in its own cgroup under /sys/fs/cgroup/bencher, with the cpuset, memory, and pids controllers enabled, and the Runner kills a Job's processes, and anything a Runner left there, through that cgroup's cgroup.kill. On an older kernel a Runner exits at startup or, started with --danger-allow-no-sandbox, fails each sandboxed Job
• Write Runner logs to stderr as JSON, and stop relaying a Job's output into runner up's log
• BREAKING CHANGE Remove --grace-period from runner up and runner run, so a Runner started with it now fails to parse its arguments. A sandboxed Job's results now leave the guest on a drive the Runner reads only once the VM has powered off, so there is no grace period left to wait out: each sandboxed Job ends about a second sooner, a guest that powers off without its results fails the Job at once rather than at its timeout, and a guest still running at its timeout has a few more seconds to write its results and power off before it is killed
• BREAKING CHANGE POST /v0/runners/{runner}/specs and bencher runner spec add now return the Runner, with its updated specs, instead of the Spec. Older CLI versions still add the Spec but cannot parse the new response, so update the CLI before using bencher runner spec add
• Show a Runner's last heartbeat and its latest status (taking Jobs or paused, why it is paused, and its disk health) on the admin Runner endpoints and in bencher system runner view
• Email server admins when a Runner's disk health changes or shows a new problem, when a Runner stays paused for more than 12 hours, and when it takes Jobs again
• Pause a Bare Metal Runner between Jobs while an md RAID array on its host resyncs, recovers, checks, repairs, or reshapes: it takes no new Job until the sync is done and never stops a running one. Start runner up with --no-raid-pause to keep taking Jobs
• Have a Bare Metal Runner take turns with host maintenance: it holds /run/bencher/job.lock through each Job, and pauses while another process holds the lock or a marker named /run/bencher/maintenance or /run/bencher/maintenance.<name> exists, so touch /run/bencher/maintenance drains it. runner run waits for the lock too. /run/bencher is now root's alone
• Report a Bare Metal Runner's disk health with each ready and paused: its md arrays at every poll, and its NVMe drives' SMART / health logs at most hourly, which needs root
• Record the project key that creates a Report, which the Report now names in project_key
• Deleting a Project now requires the Maintainer role, which Organization Leaders and server admins hold
• Add --threshold-metric and --threshold-parameters to bencher run and bencher report create to declare the metric and the parameters a Threshold checks
• Add optional metrics, hidden, focus, and layout fields to plots, so a pinned plot keeps the metrics it draws, the lines it hides, the line it focuses, and how it lays out its measures; set them with POST /v0/projects/{project}/plots, and change them, or clear them with null, with PATCH /v0/projects/{project}/plots/{plot}
• Filter the Reports list by adapter and alerts=active, and count each Report iteration's lines
• Cap a project at 64 plots: POST /v0/projects/{project}/plots refuses another with a 400 until one is deleted
• Add --bmf-version to bencher project update, so a server admin can move a Project to BMF version 1
• BREAKING CHANGE Security fix for URL validation: a project URL must now use http or https, an existing project URL that does not is shown as plain text, and the Console no longer links to a URL with any other scheme
• Security fix for the Console's embed snippet: it now escapes the project name and title
• Add Bencher to the Claude connectors directory
• Every Project, at either BMF version, now opens in the new Console at the Console's own paths: the classic Console's Project, Organization, and account pages are removed and their links redirect, and the public projects list is removed, with /perf redirecting to /explore