bench 0.8.1

A pre-release of the Rust bench CLI, built from bench-rs at ba5af628. It is not ready to replace BenchFlow 0.8 yet: some of 0.8's commands and backends are not ported (docs/rust/status.md).

curl -fsSL https://raw.githubusercontent.com/benchflow-ai/benchflow/bench-rs/install.sh | BENCH_VERSION=0.8.1 sh

Without BENCH_VERSION, the install script installs only a bench release that is not a pre-release. There is none, so it stops and asks you to set BENCH_VERSION. More in docs/rust/install.md.

What changed since 0.8.1-rc.2

• A separate verifier gets only the agent's checked outputs. This is a security fix.
  • The problem: in rc.2 and earlier, an agent that ran as root, or could become root, controlled the program that packs its outputs. It could put files outside the task's declared outputs into the separate verifier's sandbox (a reward.json, a replaced python3, a setuid program) and change its grade.
  • What is kept: before the outputs reach the verifier's sandbox, bench rewrites the archive on your machine. It keeps only regular files and folders inside the task's declared outputs, and hard links between those files. Owners and modification times are kept; a time before 1970, or 2^33 seconds or later, is written as 0. Setuid, setgid and sticky bits are cleared.
  • What is dropped: everything else, and files and hard links past 1 GiB plus 64 MiB in all, each name counted at its file's full size. The trial is still graded. Its custody.json counts every dropped entry and lists the first 50 (outputs_refused).
  • When the trial is not graded: an outputs archive with more than 100,000 entries, or with a long-name or PAX header of 64 KiB or more, is the sandbox's error (R103).
  • The verifier's log folder is emptied after the outputs are restored.
  • The same budget applies when outputs/, a stage's outputs and the agent's session are unpacked on your machine. --save-outputs saves the same checked copy.
  • Regrades: bench regrade and bench run --diff use the same check. Saved outputs that hold anything it drops (an older bench saved them unchecked), or more than 1 GiB plus 64 MiB, are not regraded; --diff runs those trials again. Honest hard links are now restored.
• Every copy out of the sandbox has a size cap, an entry limit and a 3-minute deadline. This closes rc.2's known limit, where a root agent's leftover processes could fill your disk or hang a trial through these copies.
  • Caps: 1 GiB plus 64 MiB for an archive (the outputs, a stage's outputs, the agent's session), 1 GiB of content for the verifier's folder (/logs/verifier), 16 MiB for a piece of the rubric judge's evidence, and 64 MiB for Terminus's pane log.
  • Entry limits: 500,000 entries for the verifier's folder, which always fits 100,000 files, links and folders, and 8 for any one file. Each long-name or PAX header is capped at 64 KiB.
  • What arrives: a copy that fails, or passes a cap, a limit or its deadline, is removed whole. An archive that arrives whole is made readable, and a file sent twice is replaced.
  • Copies the grade needs: the outputs of a separate verifier or a stage, and the rubric judge's evidence. When one of these fails in a way code inside the sandbox can cause, the trial is the sandbox's error (R103) and is never graded. That covers a replaced or missing tar, a broken archive and a copy that hangs. The same rule now covers files bench puts into the sandbox after the agent's turn. Such a trial runs again once, then is kept as an error. A hang, a failed evidence copy or a failed upload runs again at once; the others run again when you resume the job.
  • The verifier's folder: past its limits, or with a missing or broken tar, it is the task's error (R301) after one more try. A copy of it that hangs is the sandbox's (R103).
  • Evidence past its cap is graded as unreadable.
  • Copies that are only dropped: the agent's session, --save-outputs' outputs and Terminus's pane log. Each failure gets a line in the log, and the trial is graded; with --save-outputs, that trial then cannot be regraded.
  • When it is yours (R107): only when your own disk, quota or permissions fail, or the job folder is gone. On Docker with no [sandbox] disk limit, a sandbox that fills the disk it shares with your machine during a copy also shows as R107.
  • Details are in docs/rust/testing-tasks.md.
• The trainer files remove more of a private key an agent reads in parts.
  • Which files: each trial's trainer/verifiers.jsonl and trainer/adp.jsonl, and the job's verifiers.jsonl and adp.jsonl.
  • What is removed: when an agent shows a key with head, sed -n and tail, these files now remove the body lines of a part that starts within 8,192 characters of one of the key's markers. The parts can come in any order once one with a marker has been shown, and the rule also holds for agents that print a terminal escape or a label before their output.
  • Also redacted now:
    • OpenPGP private key blocks, whole or the half one output shows;
    • the half of a PEM or OpenSSH key when armour header lines, a grep file name with +, @, ,, %, = or non-ASCII characters, or an agent's output prefix comes before its body. rc.2 already redacted a plain half.
  • Records stay whole: only the messages' text changes. In rc.2 the job's two files cut everything between a key's BEGIN and its END, messages included, when the two were at most 8,192 characters apart, and each trial's trainer files kept the middle part.
  • The cost: in those 8,192 characters, lines or quoted strings of 16 or more base64 characters and nothing else are removed too, such as a commit hash or a path like /usr/local/bin/python3. That also happens after a key marker that stands alone in other output, such as a source file's END_MARKER constant.
  • Known limits: listed in the Trust section of docs/rust/README.md.

Known limits

• Root agents. A task that runs its agent as root, or whose image lets the agent become root, can leave processes running after the agent's turn. BenchFlow kills only the non-root agent user's processes.
  • No longer possible: through a copy out of the sandbox, those processes can no longer fill your disk, hang a trial, or put files outside the outputs into a separate verifier.
  • Disk: on Docker, a task with no [sandbox] disk limit still leaves the daemon's disk unbounded, as before. When that disk is yours, they can still fill it.
  • Failed copies: they can still make a copy the grade needs fail. That ends the trial as the sandbox's error, or as the task's (R301) for the verifier's folder, never as the agent's 0.
  • Test-runner files: files that a root agent writes inside the task's declared outputs (a conftest.py, a .pth file) still reach a separate verifier.
  • Run tasks you don't trust with a non-root agent that has no way to become root.
• The verifier's image. These apply to every task with a separate verifier, whoever the agent runs as:
  • a symbolic link in the verifier's image at, above or inside an output folder is followed when the outputs are restored;
  • outputs that clash with files the image already has can make the restore fail. That is the sandbox's error, not a grade: it is not run again in the same run, and resuming the job runs it once more, then keeps it.
• Slow copies. An honest copy must finish within 3 minutes, which for 1 GiB is about 6 MB/s. Local Docker copies are normally much faster, but no 1 GiB copy has been timed on Docker or on Daytona.

Builds: macOS on Apple silicon (aarch64-apple-darwin), and x86_64 Linux, either static (x86_64-unknown-linux-musl, the one the script picks) or with glibc 2.34 or later (x86_64-unknown-linux-gnu). Each .tar.gz has a .sha256 next to it.