v0.6.0

Added

• rugo as an eighth subject, wired the same way every other engine is: an entry in config.jsonc, a CacheKind variant, its flags, and a test asserting the four things it is started with. Nothing about the measurement path knows which engine it is running, so nothing else needed changing. This makes a sweep about a fifth longer: 23808 runs at the reference profile against 20832.
• cache-bench mem, and the cb-mem crate under it, which measures what an engine costs to hold a known number of keys. The original measures no memory at all and neither did this, so half of what somebody comparing two cache servers wants to know had no metric here. It starts each server, notes what it held before a single key went in, writes a known number of distinct keys, lets it settle and reads the largest resident set it ever had.
• The key count is known rather than estimated. memtier reports operations, and a SET pass that writes the same key twice leaves one entry behind, so a bytes-per-entry figure taken over operations is wrong by however much the pattern overlapped. The pass is sized so the clients divide the key range evenly and each writes its slice exactly once, and a count that does not divide is refused with the multiple to pick rather than rounded. Nothing asks the server how many keys it has, because DBSIZE and stats curr_items are two different questions and an engine-specific denominator is the special handling the fairness rules exist to prevent.
• Two numbers rather than one, in the file and in the generated README, which says in as many words that they are different claims. Total bytes per entry is what a machine has to have; overhead bytes per entry is what a design controls. At a hundred-odd bytes of payload per key an index that got twice as small halves the second and moves the first by a few percent, so a memory claim quoting one of them has picked the flattering number.
• The baseline each server held before any keys is recorded beside the peak and not subtracted from it, and Garnet and Dragonfly each carry a sentence saying part of their peak is a configuration rather than a consequence of the keys. Subtracting would have hidden which engines those are instead of showing it. That is D23.
• A publishable flag on a profile, false in the new smoke profile and true everywhere else including in every profile written before the flag existed. docs refuses to write a results README from a profile that is not publishable, and nothing stops the sweep, because the whole use of such a profile is to sweep it often. That is D24, and it is the difference between whether a change helped, which is answered on whatever box is free by comparing a run against the run before it, and how these engines compare, which is what a results README reads as the answer to. smoke is two thread counts, pipeline one and ten, three runs and a tenth of the reference's operations: 96 runs against the reference profile's 23808, which is the difference between minutes and days.
• The first published results, in results/wsl32coarse. Eight engines at three thread counts and four pipeline depths, five runs a cell, 480 runs, measured over a day and a half on a 32 core desktop class box with no hardware PMU, so 146 charts and none of them cycles. It is a draft rather than the reference matrix and it is published as one, because a draft is what finds the layout bugs and the provenance holes, and finding them in a sweep that took a day is better than finding them in one that took a week. Its NOTES.md says which of its bars should not be quoted, which is yo's and rugo's at eight and sixteen threads: yo here is 0.3.26, one release before the accept fix, and both engines vary by 0.4 to 0.85 run over run in those cells where Redis, Valkey and Dragonfly vary by 0.00 to 0.02.
• A NOTES.md in a results directory, linked from the generated README above the first chart when the file is there. Everything in that README is derived from the data beside it, which is what makes it trustworthy and also what makes it unable to say the one kind of thing that matters most: what is true of this sweep and of no other. A build of an engine that turned out not to be steady, an engine that had to run over TCP on that box, a window where the machine was doing something else. That is a sentence somebody has to write, it cannot be generated, and until now there was nowhere for it that a reader following a chart link would ever see. The link goes above the numbers rather than under them, because a reader who has already read the bar has read it wrong.
• cache-bench spread, which reads back the coefficient of variation every chosen file already carries and says per engine how bad the worst cell was, how many cells were over the line and which thread counts those were at. --check fails instead of only saying so, and --over moves the line, which defaults to 0.05. Reviewing the spread before publishing was a step somebody did by eye at the end of a week long sweep, which is the worst moment to be asking a person to be careful, and the hand written notes in results/wsl32coarse are what that review produced the one time it was done. Run against that directory this reproduces them: yo and rugo worst at 0.85 and 0.62, Redis and Valkey at 0.01 and 0.02 with nothing over, Garnet and Memcached in between. A cell is judged on the worse of its two passes, because a GET chart drawn from a cell whose SET pass was a coin toss is still drawn from a disturbed cell. It throws nothing away and re-runs nothing, since what to do about a noisy cell is a decision with a machine and a deadline in it.
• cache-bench archive, which packs a results directory's run files, the load generator and server logs under them and the small files beside them into one gzipped tar that can be attached to a release. The .gitignore has said since the beginning that the raw data ships as a tarball rather than in the tree, and until now nothing made that tarball, so the evidence for the one published sweep existed on the desktop box that produced it and nowhere else. That box has rebooted three times in two days. The archive carries a MANIFEST holding the SHA-256 and the byte count of every member, and --check recomputes all of them, which is a different question from whether the download finished: a checksum over the whole file says the bytes arrived, and this says the archive holds the number of runs it claims. Given --dir as well it also asks whether every chosen file the committed output.json was built from is present, which is what ties the asset to the charts. Nothing is held in memory except one file at a time, because a reference sweep's logs run to gigabytes. On results/wsl32coarse that is 4708 files into 11 MB, and a check of it takes a tenth of a second.
• cache-bench recheck, which applies the checks this build makes to a sweep that was measured under older ones. Every check in the parser is a function of the bytes memtier wrote, and those bytes are kept: all three passes of every run sit in logs/ next to the run file they produced. So when a check is added, the directories measured before it existed can be judged by it without measuring anything again, and the alternative to that is a week of somebody's box or a published directory that nobody ever goes back to. It reads each run's logs through the same parser a live run goes through and prints which runs no longer pass and what refused them; --apply moves those into runs/refused/ and records them in failures.json exactly as the day would have; --check fails if any run no longer passes, which is what a CI job wants. Refused runs are moved rather than deleted, because a run a check refuses is also the evidence for why it was refused. The runs above a refused one are renumbered, because a cell stops at its first missing number and a hole where run two was would throw away runs three, four and five along with it, and those passed. What it does not do is make a cell whole again: a sweep that met the check on the day would have retried the cell and either got its runs or given the engine up, and neither is available after the fact, so a cell that comes out of here under sampled says so rather than pretending otherwise. Run against the first published sweep it refuses 83 of 480 runs, all of them in yo, rugo and Garnet cells at eight and sixteen threads.
• A rugo part in tools/provision/install.sh and a RUGO_REF in versions.env. It is the one ref there that is a branch rather than a tag, which is written down in both places along with what it costs and when it stops being true.
• A second published results directory, results/epyc8coarse. Eight engines at three thread counts and four pipeline depths, five runs a cell, 480 runs measured over a day on an 8 core host against yo 0.3.28, the first of the two hosts with a hardware PMU. It is the directory that has the multithreaded numbers the other one lost: it offers 64 connections over four load generator threads where wsl32coarse offers 256 over sixteen, so the check that took 83 runs there takes fourteen here and no engine loses a whole column. What it costs is quiet. Ninety one of its ninety five plotted cells vary by more than a twentieth from run to run, against eighteen of seventy eight on the other box, and the runs of a cell are minutes apart rather than days, so that is the machine and not the sampling. Its NOTES.md says what can be read off a directory that loose, which is a ratio inside one engine repeated at four pipeline depths, and not a rate. On that footing it says two things about yo: its GETs go up by 1.26 to 1.46 from one I/O thread to four where Memcached, Pogocache and rugo go up by 1.83 to 2.57, and its SETs do not respond to threads or to pipeline depth at all, sitting between 111 and 140 thousand a second across every cell from pipeline ten upward while Garnet reaches 905 thousand in the same cell.

Changed

• The two chart indexes now say where their numbers came from. A link to LINEAR.md is the link that gets shared, the README is the document nobody following it opens, and what they landed on was 77 pictures of bars with no host, no method and no caveat anywhere above them, which is how one bar becomes a general statement about an engine. Each index now opens with the CPU, the logical CPU count, the distribution and the profile that was run, the line saying this is a port and these are not the original's numbers, a link back to the README and to NOTES.md where there is one, the sentence about the x axis being server I/O threads rather than clients, and the sentence saying what these numbers may not be used for, which is the same string the README carries. In golden mode there is no host named and no caveat printed, because a document generated from the spec has no measurement behind it and a caveat printed where there is nothing to caveat is what teaches people to scroll past caveats.
• A bar colour is now looked up by which server it is rather than by where that server sorted among the servers in the sweep, which is D22. The original's rule held for yo by luck, because yo sorts after all six of its names; rugo sorts between redis and valkey, and under the original's rule adding it would have silently repainted Valkey, Pogocache and yo in every chart. No published number moves and the 154 golden charts are unchanged, which is what proves it. A results file naming a server this build has no colour for is now refused, naming the server, rather than drawn in a colour a reader would take for a result.
• YO_REF moves from v0.3.26 to v0.3.27, which is the first yo release with the accept fix in it. The v0.3.26 sweep on the large host came back with yo's multithreaded cells varying by 0.40 to 0.77 run over run, against 0.00 to 0.02 for four of the six rivals measured in the same cells on the same box, and with sixteen threads slower than eight. Every worker held the same listener in its own poller and drained the whole backlog into itself, so a benchmark that opens all its connections at once had its thread split settled by one race. Nothing published moves on this bump, because no number out of those cells was steady enough to publish.
• choose no longer reduces a cell of fewer than three runs. Three is where the four files a cell comes to become four different claims: over two runs the median is one of them and the best and worst are both of them, so three of the four files carry two numbers and the spread a reader is shown is the whole sample; over one they are four copies of one number wearing four names. This never fires on a full sweep, where every cell has 31 runs, and it does not fire on the original's files either, so nothing published moves. What it is for is a directory that has lost runs since it was measured, to a sweep that stopped partway or to recheck. Such a cell is named and skipped and the count at the end says how many there were, so a chart with a gap in it is a cell that had nothing worth plotting rather than a bug in the chart layer.
• archive packs runs/refused as well, counted on its own line rather than folded into the run count. recheck moves a refused run there rather than deleting it, on the grounds that it is the evidence for why it was refused, and an archive that left it out would mean the only directory that can settle an argument about where the threshold sits is the one nobody has a copy of.
• A results README now says how many cells are plotted when that is fewer than the profile asked for. The methodology bullets are generated from the profile, which is the shape of the sweep that was wanted, and a reader takes them for a description of the data. Those are the same thing on a sweep that finished and they are not the same thing on a directory recheck has been through, so where they disagree the README says how many cells are there, the two reasons one can be missing, and that failures.json names every refused run.
• The engine count is read from CacheKind::ALL in the four places it used to be the literal seven. Those went out of step the first time an engine was added, and the only thing that said so was a test asserting a different literal.

Fixed

• The throughput of a run was read out of a memtier field that is not the throughput of the run. Ops/sec in memtier's ALL STATS is the whole completed operation count divided by Total duration, and Total duration is measured against the first of the load generator threads to finish rather than the last. Where the threads finish together those are the same number, which is why this held for three of the engines here for as long as anybody looked: checked over 1700 passes on two machines, four engines never had a pass where the last thread ran more than 9 percent longer than the first. Where they do not finish together it is not a small error. One measured pass had a thread finish after 1.387 seconds while the last took 17.682, and 25.6 million operations divided by the first of those came out as 19.4 million operations a second, higher than any rival's number anywhere on that box. Nothing downstream could see it. The count was right, the latencies were right, and it drew as the tallest bar in the chart. A pass whose last load generator thread ran more than a quarter longer than its first is now refused the way a pass that lost connections is refused, sweep records it and carries on, and the quarter is set from the same measurement: the steady engines top out at 1.09 with one reaching 1.21 on passes under two seconds, and the passes that go past a quarter go to 1.4, to 5, and in the worst case to 12.75. It is refused rather than recomputed, because a run where most of the threads were still working after one had stopped was not offering the load it was asked for over most of its length, and there is no single honest number to put in its place. That is D25, and it is the reason results/wsl32coarse has cells with no bar in them. It needs memtier 2.4.4 or newer, which is where the per thread timings it reads come from, and a file without them is an error naming the version rather than a check that quietly passes.
• A results directory stayed locked after the machine it was being written on rebooted. The lock is a file naming the pid that took it, removed when that run ends, and a run that never ends never removes it. The desktop box reboots whenever its owner wants it back, so the sweep there came up, found a lock naming a pid from before the reboot and refused to start, and did that on every retry for three hours until somebody deleted the file by hand. A lock is now taken over when the run that wrote it is provably gone, and provably is doing real work in that sentence: the file records the boot it was written on and when its pid started as well as the pid, and all three have to say the same thing before the lock is called stale. A lock from an earlier boot names nothing. A pid that is not in /proc is not running. A pid that is there but started at another moment is a different process wearing a number the first one gave back. Anything that cannot be read is treated as a no, so a machine with no /proc obeys every lock it finds, which is what every machine did before. The takeover is printed rather than done quietly, and a lock somebody is holding is refused exactly as it was.
• A sweep measured whatever else was on the machine along with the engine. doctor asks whether the box is busy once, before anything starts, and a sweep runs for days. The eight core host turned out to be a build runner as well: the sweep started on an idle machine at a load average of 0.32 and spent the next several hours against eight compilers, taking a run every twenty minutes instead of every three and producing a directory of numbers that are two workloads fighting rather than one engine measured. Every cell is now checked first, and the sweep waits rather than measuring while somebody else is there.
• The check could not be the load average, which is what the journal has recorded from the start. A load average is decayed over the last minute and a run drives every pinned core flat out for minutes, so the reading taken between two runs is mostly the run that just ended and an idle machine reads as eight busy cores. There is no threshold that separates that from eight cores belonging to somebody else. The kernel's CPU counters are cumulative instead, so a difference across a window that begins after the last server was killed is what somebody else did during that window and nothing before it. That number now goes in the journal next to the load average, and a journal written before it existed still reads.
• The load generator was the one binary in config.jsonc named for PATH to find rather than by path, which is what the original does. The original is run by hand from a login shell and this is not: a sweep is started over ssh or from a script, that shell reads no profile, and the symlink the provisioner puts in ~/.local/bin is therefore invisible. doctor --deep on the large host reported that the load generator did not answer --version while the same binary answered it fine from a prompt. It is now named the same way the eight servers are, relative to the checkout, and a test says no path in that file is a bare name.
• YO_REF was pinned below the release that added yodb serve --threads, so yo was started with a flag it rejects and exited before it listened. Every other engine here is given a thread count and yo was too, against a binary that had no such option. Found on the first real measurement rather than by reading, which is what doctor --deep is for and what it would have caught had the machine been quiet enough to run it.
• A test in cb-memtier failed on CI about one run in ten with Text file busy. It writes a shell script and runs it, and a fork in another test's thread copies the write handle it is still holding, so the exec lands in a window where the kernel will not run the file. Writing a script and starting a process now take the same lock, which is what makes the two never overlap.
• cache-bench mem defaulted to ten million entries, which does not divide by the 256 clients the reference and wsl32 profiles run, so the command refused its own default. The default is now derived from the profile: ten million rounded down to something its clients divide. A count given on the command line is still refused rather than adjusted.
• doctor --write refused to describe a machine that was busy, even when the runs it was being asked to describe were already on the disk. Both hosts here are shared, so a sweep that ran on a quiet box and a box that is quiet again an hour later are two different things, and the second one is not always available. Every other check in that command is about the machine and is true whenever it is asked; the load average is about this minute, and this minute is not when those runs were measured. It now steps aside when the results directory already holds runs, saying so in the checks line, and it refuses as before when the directory is empty and a sweep is about to start.
• The two times in host.json were fiction. started was whenever somebody typed doctor --write, which the code said the sweep would overwrite and no code anywhere did, and finished was set by nothing at all, so every generated README said the sweep was still running and always would. Both now come off the run files, which each carry the time they started: the first and last of them. That is right whichever order the commands are run in, which matters because writing this file before a sweep and writing it after one are both ordinary things to do, and it stays right for a sweep filled in over several days by several sessions, which is what an eight day sweep on a shared box actually is. The README rows say First run started and Last run started rather than naming the sweep, because a run file records when a run began and there is nowhere it records when it ended, and a row claiming the sweep finished would be claiming a fact nothing here holds. A directory with no runs in it yet still gets the clock and no finish, since that is the intended order and there is nothing better to read.
• doctor --write refused to record a machine with no frequency governor, which meant a sweep could run on the large host and then never be published from it. The same command that had just said the machine can run the profile then said it could not describe it. A missing governor is not a machine declining to answer, it is a kernel with no cpufreq driver, which is what a guest is, since a guest does not pick its own frequency. host.json now carries that as its own case beside the name a governor has when there is one, spelled out in the generated README, in the same shape the PMU field already used for the same reason. Every other fact in that file is still refused when it is missing.
• A results README said cycles per operation was measured on any host with counters, whether or not the sweep attached them. The line came off host.json alone, which answers whether the machine can count and not whether anybody counted, and the two are the same answer only on a host with no PMU, which is the only host there had been. On the first host with one the README carried the bullet saying cycles per operation was measured about two screens above the section saying the cycles chart was not drawn. The bullet now reads the profile as well and says the counters were never attached, the note where the chart would be says the same thing in its own words, and the hardware row says what the hardware has rather than what was done with it. Nothing in the published wsl32coarse documents moves, since that host has no counters and the sentence about it was already the right one.
• The caveat that travels with every results README named 256 connections whatever the sweep offered. It is duplicated word for word out of docs/methodology.md on purpose, so that the sentence about what these numbers may be used for cannot be separated from the charts, and the reference profile it was written for offers 256. A draft profile that offered 64 was publishing a claim about a load it never applied, in the one paragraph in the document whose whole job is to be narrower than the reader wants. The connection count is now substituted into that one sentence from the profile, with the copy in the methodology still checked against the constant so the two cannot drift, and a test that the substitution still finds its target if somebody rewords the sentence.
• The release workflow published eight crates and this workspace has nine. cb-mem arrived in this release and nothing added it to the list the publish step walks, so cb-docs and cache-bench would have been pushed to crates.io asking for a version of a crate that was not there, and a version on crates.io cannot be replaced. It is in the list now, between cb-stats and cb-docs, which is where its one dependency puts it. Caught by cutting this release rather than by the workflow, so the list moved into the check job that already compares the tag, the manifest and the changelog, and that job now refuses a release whose publish order and workspace do not name the same crates. Everything under crates/ is published and xtask is not, so the directory listing is the set it checks against.
• install.sh checking out a branch ref gave the commit the last run left behind, forever. git fetch origin moves origin/main and leaves the local main alone, so the build was skipped as already current. It now prefers the remote-tracking name where there is one, which resolves identically for every tag already pinned.