v0.4.0

• Add multi-process scenarios, whose roles, declared in [[package.metadata.base-benchmark.scenario.role]], are each one supervised process in its own process group for every sample: services, such as a server, and one driver, such as a load client. Exactly one role runs the subject, with the scenario's arguments and variants; the others run the same program with the same arguments for every subject. Each sample is one session: every executable is verified, services start in declared order and are each waited on until they print a line matching their ready pattern, whose {NAME} captures, such as the address a server bound, later roles receive as {ROLE.NAME}; then the driver runs, and the services are stopped in reverse order, by a declared signal with a grace period after which the group is killed, or by waiting for them to exit by themselves. Ready patterns match in time linear in the line's length, earlier captures taking the longest text that lets the rest of the line match. In role arguments, {{ and }} are literal braces, and any other brace, or a reference to a value no earlier service captures, is an error when the scenario is planned. Every role runs under the same launch controls as a single-process workload, one deadline covers the session, and a failure, the deadline, cancellation, or a panic kills every role's group and reaps every role, in reverse start order, before the host lock is released. A service that exits before it is ready or stopped, does not stop successfully, or a driver that fails or writes an invalid record stops the run with an error naming the role. The driver's record is the sample's.
• Add external programs, such as a load client installed with cargo install, which roles name with external = "NAME" and runs give by their path on the measuring host with --program NAME=PATH (Workspace::programs, Workspace::with_program). The plan records each program's resolved path and SHA-256 (Plan::programs, ExternalProgram), and the hash is verified before every sample, so a program changed during a campaign fails it. --host passes program paths to the host unchanged.
• Add --role-cpus ROLE=CPUS (RunRequest::role_cpus) to pin each role to its own CPUs. Recorded campaigns need a CPU set for every role, and --cpus applies only to single-process scenarios.
• Start every workload and role with the stop signals, SIGTERM, SIGINT, SIGHUP, SIGQUIT, SIGUSR1, and SIGUSR2, at their default dispositions and unblocked, even when base-benchmark started with some of them ignored, as & in a non-interactive shell leaves SIGINT and SIGQUIT and nohup leaves SIGHUP, or blocked, as a launcher's signal mask passes to what it starts. A child keeps the signals its parent ignores and the signal mask of the thread that started it, so a service stopped by one of them would outlast it and be killed after its grace period. Other signals' dispositions and the rest of the mask are left as inherited.
• Cancel campaigns and preparations on the first SIGINT, SIGTERM, or SIGHUP even when base-benchmark started with them blocked: the handler's thread unblocks them, so they reach it. They stayed pending, so neither Ctrl-C nor SIGTERM stopped base-benchmark.
• Kill and reap a single-process workload before the host lock is released when a panic, or a helper thread that fails to spawn, interrupts running it, as sessions already did. It was left running and unreaped after the lock was released.
• Add the driver's measured window. Under the runner, a multi-process scenario's driver marks when its counted operations start and stop with the lines base-benchmark window start and base-benchmark window stop on stdout, as whole, flushed lines before any output of its record; it is told to by BASE_BENCHMARK_CONTROL=stdio, which the plan records as the driver's PlannedRole::environment, and the runner acknowledges the start on its stdin once it has snapshotted every service, then closes its stdin. Window, available without the runner feature, sends them: Window::from_environment returns None outside the runner, Window::start_acknowledged waits for the acknowledgement, so that the snapshots precede the first counted operation, and Window::start does not. A driver that sends no markers, sends one twice, stops before it starts, sends an invalid marker, a marker after output of its record, or one without a newline at the end of its output fails its sample, as does one whose output a descendant holds open after it exits. Window implements Debug.
• Measure each service's CPU time over the driver's window, from its CPU-time clock, and the kernel's user and system split, from /proc/PID/stat, at the window's markers, as the cpu_time, user_cpu_time, and system_cpu_time totals of the new Phase::Window, journalled with each service's RoleEntry::window. Summaries report them as ROLE/METRIC totals, and each divided by the driver's operations as ROLE/METRIC_per_operation, CPU per request, with the new Statistic::ApproximatePerOperation, reported as ns/op, approximate: the window's edges, failed requests, and draining after the stop marker are not exactly the counted operations. No other service value is divided. The time counted is what the kernel accounts to the service's own threads: its child processes, such as the workers of a forking server, and softirq time spent on its behalf, such as network processing, are not included, so CPU per request understates a network server's cost, and summaries warn that window CPU time covers only each service's own process.
• Treat invalid responses as making comparisons invalid. A driver role names the total in its record that counts responses which failed its validation with invalid-responses, such as base-benchmark-load's responses_invalid (Role::invalid_responses, PlannedRole::invalid_responses). In screening, a sample with any keeps its numbers, and the summary reports every comparison involving its subject as invalid, without metrics (PairSummary::invalid), with a warning listed first; a recorded campaign fails on any. A driver record without the declared total fails its sample. Summaries of multi-process scenarios state that operations are the driver's successful operations, so every per-operation value, the driver's own included, and every request error count, is per successful operation.
• Add helper roles, which run a workspace binary, such as a load client or a wrapper of a separately published one, with binary = "NAME" and optionally package and features. Preparation builds each helper the prepared scenarios need once, from the baseline tree, unsalted, and records it as a fixed artifact in the manifest's helpers (HelperArtifact, ArtifactRole::Helper, PrepareOptions::helpers, Helper); every subject runs the same executable, staged under a name of the same length as the artifacts'. Workspace::smoke_executable skips scenarios whose roles run helpers, since none was built.
• Record each external program's version, the first line it prints for --version within 10 seconds, on stdout or else the start of stderr (ExternalProgram::version), or none when that fails; an unknown version is recorded, not refused. Campaigns ask once they hold the host lock, on the CPUs of the role that runs the program and under the campaign's cancellation, and record it in the plan in metadata.json; plans themselves leave it unknown.
• Breaking: artifact manifests are version 3, adding helpers; version 2 manifests, which have none, are still read and run. ArtifactManifest and PrepareOptions gain public fields, and ArtifactRole gains Helper, which breaks struct literals and exhaustive matches.
• Check whether roles share physical cores: two roles share one when their CPU sets include the same logical CPU or SMT siblings, as the measuring host reports them in /sys/devices/system/cpu/cpuN/topology/thread_siblings_list (Topology, PlanOptions::topology). Sharing that the scenario does not declare is refused by recorded campaigns and warned about by screening campaigns, when they start and in their summaries, and so are placed CPUs whose siblings the host does not report. A scenario that sets shared-placement = true, for example to model contention on a cloud host, may share cores in both modes, through reported siblings or not, and its summaries and reports label its placement as deliberately shared (Summary::placement). A virtual machine that hides SMT cannot be checked. Roles placed on CPUs that do not exist or are offline, according to /sys/devices/system/cpu/present and online, fail when the campaign is planned. Plans record the sharing they found (Plan::shared_cores, SharedCore, Plan::unknown_topology).
• Breaking: Phase gains Window, which records cannot contain, so records remain version 2; matches on Phase need the new arm. Summaries are version 3, since they can contain window metrics and the approximate statistic.
• Breaking: plans are version 3 and campaigns version 2. Plans of scenarios with roles add roles (PlannedRole) and programs, and their samples add roles (RoleEntry), journalling each role's command, CPUs, stack padding, executable hash, host, ready line and captured values, exit status, whether it was killed after its stop grace period, and the end of its output; plans and samples of single-process scenarios omit them and are otherwise unchanged. Scenario, PlanOptions, RunRequest, Workspace, and SampleEntry gain public fields, which breaks struct literals of them. Workspace::smoke needs the programs of every scenario's roles.

v0.3.0

• Read Cargo metadata for path dependencies again with network access when reading it offline fails, still locked, saying so on standard error first. Building only the workload package does not download packages that only other workspace members need, so on a host with a fresh Cargo cache every artifact recorded its path dependencies as unknown, which was noticed only when a recorded campaign refused them. Preparation now warns when metadata still cannot be read, naming the tree and giving Cargo's errors from both attempts. Metadata is read once per tree rather than once per feature set, not again with network access once preparation is cancelled, and in the tree, as builds are, so that its Cargo configuration, such as source replacement or net.offline, applies; it was read in the current directory, as was the metadata --host reads to find path dependencies to mirror.
• Don't copy a remote command's output back when the command failed before creating it on the host. rsync's errors about the missing directory followed, and buried, the host's own error. Outputs that exist, such as those of failed or cancelled campaigns, still come back.
• Name every metric with the same hazard in one comparison or subject in a single warning, and order warnings by hazard: build layout first, such as a variant comparison whose sign reverses in the layout replica, then metric definitions, campaign controls, and last, metrics that changed in both directions. A screening run of two backends across two builds and a replica gave 48 warnings, 34 of them one line per metric that moved both ways; it now gives 8. The warnings' wording changes; the statistics, and which hazards are reported, do not.
• Fail a preparation cancelled while a build runs with an Interrupted error, as one cancelled between builds does, and write no manifest. Cancellation was checked only before each build, so when Cargo finished the build that was running, as it does when only base-benchmark is signalled, preparation succeeded; if it was the last build and the tree's Cargo metadata could not be read offline, it was not read again online, so its path dependencies were recorded as unknown without a warning, and a recorded campaign later refused the artifact without saying why. When Cargo, or a program a build runs such as rustc -Vv, fails once preparation is cancelled, as Ctrl-C makes it fail by reaching the whole process group, preparation also fails as cancelled, with that failure in parentheses, as in preparation was cancelled (current: cargo build failed with signal: 2 (SIGINT)), rather than as failed. Likewise, --host reports a command as cancelled, rather than failed, when rsync or another program it runs fails once it is cancelled: cancelled before the host's command started, with that failure, when the command had not started.
• Write frozen executables, staged copies, and the layout salt wrapper through a child process, cat run by /bin/sh, so that this process never holds a file it will run open for writing. A process forked meanwhile by another thread, such as when base-benchmark is used as a library by a multithreaded program, shared the writable descriptor until it ran another program, and perf running a staged copy, or Cargo running the wrapper, could then fail with ETXTBSY; under full CPU load, 13 of 180 runs of the library's tests failed this way, and none of 360 since. Programs the runner starts itself already retried. Copies of trees for layout replicas and overlays are still written in this process.
• Breaking: state overlaid builds in reports: the overlaid package, its revision and state, the artifacts built with it, and each artifact's lockfile changes. Summary gains a public overlays field (OverlaidArtifact) with the same, which breaks struct literals of Summary. It is omitted from summary.json when nothing was overlaid, so other summaries are unchanged and remain version 2; 0.2.1 reads Summary with unknown fields denied, so it rejects summaries of overlaid campaigns.
• Breaking: Summary, SubjectSummary, MetricSummary, Distribution, PairSummary, PairMetric, and OverlaidArtifact are #[non_exhaustive], since summaries are produced by summarize or read from summary.json rather than constructed, so that adding fields to them no longer breaks callers. Struct literals and exhaustive patterns of them outside this crate no longer compile; patterns need ...

v0.2.1

• Chain a rustc wrapper set in Cargo configuration, such as a global sccache in ~/.cargo/config.toml, when salting layout replicas, instead of refusing to prepare them. The wrapper is found as Cargo finds it: the environment, then build.rustc-wrapper in the configuration nearest the tree, then $CARGO_HOME's, including files they include, with relative paths resolved as Cargo resolves them. An empty $CARGO_HOME counts as unset, and a relative one is relative to the tree, as for Cargo. Invalid configuration files are reported with their parse location.

v0.2.0

• Add --host HOST to prepare, run, and smoke (remote::Remote) to run them on a measuring host over SSH. Local paths, made canonical, are mirrored on the host with rsync, including the repositories of linked worktrees, the release of this version is installed there when it is missing, and outputs are copied back, also after a failure. Interrupting the local command stops the host's. --remote-directory and --remote-program choose where and what runs. Add version, which prints the command's version.
• run makes its output directory absolute, so a relative --output works when the command runs outside the workspace root.
• Build layout replicas with a distinct -C metadata salt for every crate, recorded as BuildIdentity::layout_salt. Rebuilding at another path alone gave an executable byte-identical to the baseline, since Cargo identifies workspace members by relative path, so 0.1's replicas measured run-to-run variation of one executable rather than build layout. The salt is added by a RUSTC_WRAPPER script in the replica's target directory, which chains RUSTC_WRAPPER or CARGO_BUILD_RUSTC_WRAPPER from the environment, such as sccache. A wrapper set in Cargo configuration cannot be chained, so preparing replicas then fails. RUSTFLAGS and Cargo configuration are untouched. Summaries warn about replicas identical to their baseline. Salting needs Unix.
• Add --overlay true (PrepareRequest::overlay, Overlay) to build every tree with the current workspace's workload package, so that revisions from before the workload existed, or with an older version of it, run the same harness. Each tree is copied to a sibling directory, its workload package replaced, its lockfile seeded with the current workspace's entries for packages it does not lock, and the copy built without --locked; trees are never modified. Artifacts record the overlaid package in SourceIdentity::overlay, and in SourceIdentity::lockfile_changes every locked version the build changed and every added package that resolved differently from the current workspace. Recorded campaigns require the overlaid package to be committed and refuse lockfile changes. PrepareOptions and SourceIdentity gain these public fields.
• Let scenarios set their own Cargo features, so that a diagnostic or allocation-counting build can be a declared scenario without reaching the timed builds. Features are canonical: entries are split on commas and whitespace, sorted, and deduplicated (canonical_features). Preparation builds each tree once for every distinct feature set of the scenarios it prepares, or of those named with --scenario, and a run uses only the builds with exactly its scenario's features; Workspace::smoke builds every feature set. Executables are frozen under artifacts/LABEL/N/, numbered by feature set.
• Breaking: artifact manifests are version 2, with features on each Artifact instead of the manifest, and plans version 2. A manifest's artifacts all have known features or all unknown ones, and each feature set builds the same trees with the same roles. Campaigns from 0.1 still parse and summarize; reading a version 1 artifacts.json asks to prepare again. Scenario and Artifact gain a public features field. Workspace::prepare takes a PrepareRequest, PrepareOptions has feature_sets, and Plan::new narrows the plan's artifacts to its scenario's builds (ArtifactManifest::for_features). Workspace::smoke_executable takes the features the executable was built with, skips scenarios that need others, and fails if none can run.
• Add Metric::quantile and Measurement::Quantile for order statistics such as a 99th percentile latency, summarized as Statistic::Quantile with the quantile, and reported as, for example, ns p99. A metric reported at different quantiles is a different definition, flagged rather than compared.
• Records are now version 2 and summaries version 2, since both can contain quantiles. The runner still accepts version 1 records, so workloads built against 0.1 keep working. A 0.1 runner rejects 0.2 records: with a version mismatch, or for records with quantiles, with an unknown variant quantile.
• Breaking: Measurement and Statistic gain variants and are now #[non_exhaustive], so matches on them need a wildcard arm. Statistic is no longer Eq, since it now carries the quantile.
• Verify executables without reading them again while their identity is unchanged: device, inode, size, and modification time on Unix, taken from the opened file. Copies made when staging cannot hard-link inherit their source's verified hash. Every sample and every campaign verifies its executable, so smoke runs of unoptimized test builds spent most of their time hashing; a 19-scenario smoke test of a 50 MB debug executable took 27 s. A same-size rewrite that keeps or restores the modification time, or happens within the file system's timestamp granularity, is no longer detected within one process. sha256_identified is public.
• Keep the collector's LLVM_PROFILE_FILE in the workload environment outside recorded campaigns, so that coverage runs collect workload profiles instead of leaving default_*.profraw files in the workspace.

v0.1.1

• Build the runner on every platform, so that projects whose task package depends on bake-benchmark build in CI on macOS and Windows. Collecting samples needs Linux, the only platform it is tested on: elsewhere run and smoke return an Unsupported error before creating a campaign or building anything. FreeBSD and Android, where the runner may previously have compiled untested, are now unsupported too. The host lock and replica copies of symbolic links need Unix. CI checks the library and command on macOS and Windows.

v0.1.0

• Establish the repository and shared Base development tooling.
• Record the organization-wide benchmarking audit and the two-repository Rust architecture.
• Incorporate scheduler consumer feedback on comparative preparation, layout controls, screening, and declared failure outcomes into the plan.
• Capture the lower-priority historical data, normalized comparison, and graph/analysis follow-up.
• Define a shared performance investigation record and template that retain rejected, invalid, and inconclusive attempts as well as adopted changes.
• Add versioned process records with explicit metric phases, sources, directions, and resolution, and Interval for measured-phase wall, CPU, and context-switch totals.
• Discover scenarios and variants from [package.metadata.base-benchmark], including unpublished packages and package-level defaults.
• Prepare labelled source trees into frozen, hashed executables with source and build identity, plus a same-source layout replica built beside the baseline.
• Run seeded random rounds under a host-wide lock with CPU placement, deadlines, bounded output, signal cancellation, and process-group cleanup; journal every sample and record completion state.
• Summarize per-round paired differences and resolution-aware ratios, and warn about missing placement, inconsistent directions, and changes within or reversed by the layout replica.
• Add the base-benchmark command with list, prepare, smoke, run, and summarize.
• Control launch layout: stage executables under equal-length paths, clear the workload environment to PATH and LC_ALL=C, start in the workspace root, and add seeded random stack padding to every sample.
• Build any number of layout replicas with --replicas N, and warn when variant comparisons have no replica or an unresolved one.
• Report exact two-sided sign tests, count differences below a metric's resolution as ties, show percentage changes only when every round resolves, never show counts as percentages, and compare metrics only under identical definitions.
• Add a microsecond-resolution cpu_time total, and declare the tick-sampled user and system split at 10 ms.
• Reject campaigns whose records differ in workload or signature, or whose artifacts run a variant with different parameters.
• Record cargo -Vv, build-affecting environment variables, Cargo configuration files, and the Git state of out-of-tree path dependencies; recorded mode requires all of them to match and be committed.
• Add measure-arguments for screening and recorded runs, show resolved baseline variants, and add Workspace::smoke_executable.
• Split the runner into a default runner feature so workloads link only records and Interval.
• Use the fixed, shared host lock /tmp/base-benchmark-host.lock, write metadata atomically, fail samples whose escaped descendants hold the output open, retry busy executables, and handle signals process-wide: the first cancels guarded work, including preparation between builds, and a second exits.
• Take the scenario positionally in base-benchmark run, and reject a repeated --manifest-path.
• Never follow a symbolic link at the shared lock path, open an existing lock without O_CREAT so other users can share it under fs.protected_regular, and widen permissions only on a lock file created exclusively.
• Compute sign tests in logarithms, so long campaigns do not underflow to zero.
• Record path dependencies as unknown when Cargo metadata cannot be read, and reject them in recorded mode; filter metadata to the host platform, and include ~/.cargo configuration when $CARGO_HOME is unset.
• Leave signals that were ignored at startup alone on Linux, and treat unavailable values as compatible with any statistic of the same metric.