Skip to content

v0.5.0, the runner

Choose a tag to compare

@tamnd tamnd released this 04 Sep 11:53
· 40 commits to main since this release
84e2cfe

This is the release where the project starts measuring. Everything before it was arithmetic over numbers somebody else took.

cache-bench run redis --threads 8 --pipeline 10 --perf no --run 1 measures one cell once and writes one file. Ask the binary its version, clear any leftover socket, start the server pinned to the cache half of the cores, wait for it to answer a real command over that socket, run a warmup pass and throw it away, attach perf if the cell wants counters, run the measured set pass and the measured get pass pinned to the other half, detach, stop the process group, confirm nothing survived, write. The sequence is the original's, in the original's order, and it does not vary by engine. Run it once per cell and the sweep is the loop around it, which is M6.

What is not the original's is what happens when the sequence goes wrong. Six things are refused here that the original would have measured through, and every one of them produces a plausible number rather than an error:

  • A server left over from an earlier cell still answering on the socket. The original's answer is to pkill every process whose name it recognises, including ones it did not start, which on a shared machine kills somebody else's work and still leaves a full set of real numbers belonging to the wrong engine.
  • Two sweeps sharing a results directory. That is not a race over a file, it is a race over the machine. Both pin the same cores and both bind the same socket, and the numbers come out low with nothing in them saying why.
  • A perf cell on a machine whose counters do not answer, which would write a file full of missing counters that looks exactly like a machine where one counter is unsupported.
  • More I/O threads than the profile pinned cores. Seventeen threads on sixteen cores measures the scheduler and draws a bar.
  • A profile written for a bigger machine than the one running it. A pin naming some cores that exist and some that do not is narrowed by the kernel rather than refused.
  • A memtier pass that did not complete the operations it was asked for, or came back missing a percentile.

A run that fails anywhere writes nothing, because a partial file cannot be told from a complete one by the stage that reads it next. A cell already on disk is skipped rather than measured again. That is the whole of how a sweep that takes days survives a reboot.

Also in this release: all seven adapters wired to one supervisor with no per engine special handling, readiness by protocol round trip rather than by sleeping, CPU pinning through sched_setaffinity between fork and exec rather than through taskset, a stopping sequence that confirms the process group is gone, the perf driver with counters taken over the server process only and only for the measured passes, a PMU probe that checks a live counter rather than the presence of a directory, and a strict memtier parser where the original uses a query that yields zero for a missing field.

The M5 exit condition is a gate and it has not been run. Seven servers each starting, answering, completing a run and stopping cleanly, with and without perf, needs a Linux host with all seven built on it. Everything up to that point is verified: unit tests on every piece, one whole run end to end against a cache server and a load generator that are not real, and thirteen checks on three operating systems. The gate is hardware work and it stays open until it runs on hardware.

The full list is in CHANGELOG.md.