v0.6.1

Fixed

• chart --dir drew unstamped charts when --profile was left off, and the stamp is part of the picture, so every chart came out different from the committed one. Checking a published directory that way reported that all 146 of its charts failed the manifest, which reads as the charts not being reproducible when what happened is that an argument was missing. The profile now comes off the directory's own host.json when there is no --profile, which is the only right answer for a results directory and one nobody can get wrong. A test says every published directory is named for the profile it records, since the two disagreeing would put one name on the directory and another on the charts inside it.
• The spell check failed on a Redis build id. Every engine's version line is recorded verbatim, because a chart is only worth anything if it says what it measured, and Redis 8.10.1 on the 8 core host prints a build id with a two letter run in the middle of it that reads as a misspelling of an English word. Nothing here writes those strings and none of them can be corrected, so the hash is skipped rather than the file it sits in, which keeps the prose around it checked. The first host got through on luck rather than on anything the check knew.
• Nothing ran the check that redrawing the charts from the committed output.json reproduces the committed PNGs, which is the whole reason output.json is committed beside them and is M8's exit gate written out as a sentence. CI now runs it for every published directory, in the job that already checks the generated documents, comparing SHA-256 rather than pictures. It is also the only check here that can see a change to the chart code quietly moving a pixel.

v0.6.0

Added

• rugo as an eighth subject, wired the same way every other engine is: an entry in config.jsonc, a CacheKind variant, its flags, and a test asserting the four things it is started with. Nothing about the measurement path knows which engine it is running, so nothing else needed changing. This makes a sweep about a fifth longer: 23808 runs at the reference profile against 20832.
• cache-bench mem, and the cb-mem crate under it, which measures what an engine costs to hold a known number of keys. The original measures no memory at all and neither did this, so half of what somebody comparing two cache servers wants to know had no metric here. It starts each server, notes what it held before a single key went in, writes a known number of distinct keys, lets it settle and reads the largest resident set it ever had.
• The key count is known rather than estimated. memtier reports operations, and a SET pass that writes the same key twice leaves one entry behind, so a bytes-per-entry figure taken over operations is wrong by however much the pattern overlapped. The pass is sized so the clients divide the key range evenly and each writes its slice exactly once, and a count that does not divide is refused with the multiple to pick rather than rounded. Nothing asks the server how many keys it has, because DBSIZE and stats curr_items are two different questions and an engine-specific denominator is the special handling the fairness rules exist to prevent.
• Two numbers rather than one, in the file and in the generated README, which says in as many words that they are different claims. Total bytes per entry is what a machine has to have; overhead bytes per entry is what a design controls. At a hundred-odd bytes of payload per key an index that got twice as small halves the second and moves the first by a few percent, so a memory claim quoting one of them has picked the flattering number.
• The baseline each server held before any keys is recorded beside the peak and not subtracted from it, and Garnet and Dragonfly each carry a sentence saying part of their peak is a configuration rather than a consequence of the keys. Subtracting would have hidden which engines those are instead of showing it. That is D23.
• A publishable flag on a profile, false in the new smoke profile and true everywhere else including in every profile written before the flag existed. docs refuses to write a results README from a profile that is not publishable, and nothing stops the sweep, because the whole use of such a profile is to sweep it often. That is D24, and it is the difference between whether a change helped, which is answered on whatever box is free by comparing a run against the run before it, and how these engines compare, which is what a results README reads as the answer to. smoke is two thread counts, pipeline one and ten, three runs and a tenth of the reference's operations: 96 runs against the reference profile's 23808, which is the difference between minutes and days.
• The first published results, in results/wsl32coarse. Eight engines at three thread counts and four pipeline depths, five runs a cell, 480 runs, measured over a day and a half on a 32 core desktop class box with no hardware PMU, so 146 charts and none of them cycles. It is a draft rather than the reference matrix and it is published as one, because a draft is what finds the layout bugs and the provenance holes, and finding them in a sweep that took a day is better than finding them in one that took a week. Its NOTES.md says which of its bars should not be quoted, which is yo's and rugo's at eight and sixteen threads: yo here is 0.3.26, one release before the accept fix, and both engines vary by 0.4 to 0.85 run over run in those cells where Redis, Valkey and Dragonfly vary by 0.00 to 0.02.
• A NOTES.md in a results directory, linked from the generated README above the first chart when the file is there. Everything in that README is derived from the data beside it, which is what makes it trustworthy and also what makes it unable to say the one kind of thing that matters most: what is true of this sweep and of no other. A build of an engine that turned out not to be steady, an engine that had to run over TCP on that box, a window where the machine was doing something else. That is a sentence somebody has to write, it cannot be generated, and until now there was nowhere for it that a reader following a chart link would ever see. The link goes above the numbers rather than under them, because a reader who has already read the bar has read it wrong.
• cache-bench spread, which reads back the coefficient of variation every chosen file already carries and says per engine how bad the worst cell was, how many cells were over the line and which thread counts those were at. --check fails instead of only saying so, and --over moves the line, which defaults to 0.05. Reviewing the spread before publishing was a step somebody did by eye at the end of a week long sweep, which is the worst moment to be asking a person to be careful, and the hand written notes in results/wsl32coarse are what that review produced the one time it was done. Run against that directory this reproduces them: yo and rugo worst at 0.85 and 0.62, Redis and Valkey at 0.01 and 0.02 with nothing over, Garnet and Memcached in between. A cell is judged on the worse of its two passes, because a GET chart drawn from a cell whose SET pass was a coin toss is still drawn from a disturbed cell. It throws nothing away and re-runs nothing, since what to do about a noisy cell is a decision with a machine and a deadline in it.
• cache-bench archive, which packs a results directory's run files, the load generator and server logs under them and the small files beside them into one gzipped tar that can be attached to a release. The .gitignore has said since the beginning that the raw data ships as a tarball rather than in the tree, and until now nothing made that tarball, so the evidence for the one published sweep existed on the desktop box that produced it and nowhere else. That box has rebooted three times in two days. The archive carries a MANIFEST holding the SHA-256 and the byte count of every member, and --check recomputes all of them, which is a different question from whether the download finished: a checksum over the whole file says the bytes arrived, and this says the archive holds the number of runs it claims. Given --dir as well it also asks whether every chosen file the committed output.json was built from is present, which is what ties the asset to the charts. Nothing is held in memory except one file at a time, because a reference sweep's logs run to gigabytes. On results/wsl32coarse that is 4708 files into 11 MB, and a check of it takes a tenth of a second.
• cache-bench recheck, which applies the checks this build makes to a sweep that was measured under older ones. Every check in the parser is a function of the bytes memtier wrote, and those bytes are kept: all three passes of every run sit in logs/ next to the run file they produced. So when a check is added, the directories measured before it existed can be judged by it without measuring anything again, and the alternative to that is a week of somebody's box or a published directory that nobody ever goes back to. It reads each run's logs through the same parser a live run goes through and prints which runs no longer pass and what refused them; --apply moves those into runs/refused/ and records them in failures.json exactly as the day would have; --check fails if any run no longer passes, which is what a CI job wants. Refused runs are moved rather than deleted, because a run a check refuses is also the evidence for why it was refused. The runs above a refused one are renumbered, because a cell stops at its first missing number and a hole where run two was would throw away runs three, four and five along with it, and those passed. What it does not do is make a cell whole again: a sweep that met the check on the day would have retried the cell and either got its runs or given the engine up, and neither is available after the fact, so a cell that comes out of here under sampled says so rather than pretending otherwise. Run against the first published sweep it refuses 83 of 480 runs, all of them in yo, rugo and Garnet cells at eight and sixteen threads.
• A rugo part in tools/provision/install.sh and a RUGO_REF in versions.env. It is the one ref there that is a branch rather than a tag, which is written down in both places along with what it costs and when it stops being true.
• A second published results directory, results/epyc8coarse. Eight engines at three thread counts and four pipeline depths, five runs a cell, 480 runs measured over a day on an 8 core host against yo 0.3.28, the first of the two hosts with a hardware PMU. It is the directory that has the multithreaded numbers the other one lost: it offers 64 connections over four load generator threads where wsl32coarse offers 256 over sixteen, so the check that took 83 runs there takes fourteen here and no engine loses a whole column. What it costs is quiet. Ninety one of its ninety five plotted cells vary by more than a twentieth from run to run, against eighteen of seventy eight on the other box, and the runs of a cell are minutes apart rather than days, so that is the machine and not the sampling. Its NOTES.md says what can be read off a directory that loose, which is a ratio inside one engine repeated at four pipeline depths, and not a rate. On that footing it says two things about yo: its GETs go up by 1.26 to 1.46 from one I/O thread to four where Memcached, Pogocache and rugo go up by 1.83 to 2.57, and its SETs do not respond to threads or to pipeline depth at all, sitting between 111 and 140 thousand a second across every cell from pipeline ten upward while Garnet reaches 905 thousa...

Raw runs and logs for results/wsl32coarse

The raw evidence behind results/wsl32coarse, as of the republish in #69. Not a code release. Nothing here is on crates.io and no binaries are attached.

4636 files. 709 run files under runs, which are the measurements the charts are drawn from, 83 under runs/refused, which are the ones a check refused after the fact, and 3841 under logs, which is every pass of every run exactly as the load generator and the servers wrote it.

The refused runs are here on purpose. They were refused by D25, which is memtier reporting Ops/sec as the completed operation count over Total duration while Total duration runs only to the first of its load generator threads to finish. A pass whose last thread ran more than a quarter longer than its first is now refused, and where that line belongs is a question about what it keeps and what it throws away. That question cannot be argued without the runs it threw away, so they ship, in their own directory and counted separately from the runs.

The logs are why the check could be applied at all. It arrived after all 480 runs were measured, and cache-bench recheck judged them again from these bytes rather than by measuring anything a second time. Anybody who wants to apply a different threshold to this sweep can do the same thing to this tarball.

To check it:

cache-bench archive --check wsl32coarse-runs.tar.gz

which reads every file back against the MANIFEST inside the archive.

What the numbers do and do not support is in NOTES.md next to the charts. The short version is that this is a draft sweep at five runs a cell on a host with no hardware PMU, 18 of its 96 cells have no bar left in them, and two of the eight engines lose their whole multithreaded half.

Raw runs and logs for results/epyc8coarse

The raw evidence behind results/epyc8coarse, the second published results directory and the first from a host with a hardware PMU. Not a code release. Nothing here is on crates.io and no binaries are attached.

4705 files. 466 run files under runs, which are the measurements, and 380 reduction files beside them, which are the median, best, worst and average the charts are drawn from, one set per cell. 14 under runs/refused, which are the runs a check refused after the fact. 3841 under logs, which is every pass of every run exactly as the load generator and the servers wrote it.

Fourteen refusals against 83 on the 32 core host, all of them D25, which is memtier reporting Ops/sec as the completed operation count over Total duration while Total duration runs only to the first of its load generator threads to finish. The difference between the two hosts is the load: this profile offers 64 connections over 4 load generator threads where that one offers 256 over 16, and an accept split that is badly uneven at 256 is only slightly uneven at 64. Ten of the fourteen are rugo, two are yo, two are Garnet.

What this host gives up is quiet. 91 of its 95 plotted cells vary by more than a twentieth from run to run, and the sweep waits for a quiet machine, so that is not a filtering problem. The 65 runs measured before that wait was tight enough were set aside rather than published, and comparing them against the runs that replaced them says the waiting bought nothing: across the thirteen cells measured both ways the median moved by about a tenth and it moved up as often as down.

To check it:

cache-bench archive --check epyc8coarse-runs.tar.gz

which reads every file back against the MANIFEST inside the archive. cache-bench recheck can also judge this sweep against a threshold written after it was measured, from these bytes, without measuring anything again, which is the whole reason the logs are kept.

What the numbers do and do not support is in NOTES.md next to the charts. The short version is that this directory supports ratios inside one engine and not rates.

v0.5.1

Added

• The machine half of cache-bench doctor. It reads what the kernel publishes about the box it is on, the CPU, the memory, the governor, the mitigations, the load average and whether there are counters to count with, and then measures the profile that is about to be swept against it. A profile that names more cores than the machine has, a profile that sweeps the cycles half of the matrix on a host with no PMU, a working set that would not fit in memory, or a load average that says somebody else is using the machine: each of those is refused rather than warned about, because a warning printed at the start of a job that runs for eight days is read by nobody and each of them produces numbers rather than an error.
• doctor --write, which records what the machine is in host.json next to the results. A fact the machine does not publish is refused rather than written as unknown, since this file is the whole of what a published results directory says about where its numbers came from. Nothing in it names the machine, and if a hosts.toml is there its names are checked against the file before it is written.
• doctor --deep, which starts each of the seven servers in turn, waits for it to answer, stops it and checks the group is gone. A binary of the wrong architecture, a server built without unix socket support, a Garnet whose runtime is not installed: all of them read as a correct config and all of them fail on the first run of a sweep instead of here.
• Every file the kernel publishes is parsed by a function that takes a string, with tests over the text an ARM box, an x86 box and a container each produce. A parser that has only ever run on the machine it was written on is a parser nobody has checked.
• cache-bench sweep, the loop around run. It measures engine, then thread count, then pipeline depth, then whether counters are attached, then the run number, which is the original's order and which keeps all the runs of one cell together in time, so an hour where somebody else was using the machine comes out as one cell that is visibly wrong instead of a slight tilt across every cell in the matrix. It takes the results directory once for the whole sweep rather than once per cell, and it checks that the config names a binary for every engine it is about to sweep before it starts, because a config that never named Garnet is one that fails on day three of eight having measured everything before it.
• Restart by file existence, with a file that will not parse counted as absent rather than as a run. A sweep that lost power partway through a write leaves a file of the right name holding half a run, and that file measured again is a run, while that file trusted is a truncated measurement in a median that nothing downstream can tell from a real one.
• results/logs/sweep.jsonl, one line per run attempted, in the order they were attempted, with the load average taken just before each one. Before rather than after, because the question it answers is whether the machine was already busy and a run is itself load. Nothing reads this file. It is there for the person who looks at one odd cell a week later and wants to know what else was happening at three in the morning on day six.
• results/failures.json, which names every cell that was attempted and produced no file, with the reason verbatim, when it last failed and how many times it has been tried across every sweep of that directory. A chart cannot draw a cell that is not there, and the alternative to naming it is a zero, and a zero is a claim about an engine while an absence is not.
• A failed cell no longer stops the sweep, since the other ten thousand are still worth measuring, but a sweep that did not measure everything says so on the way out rather than reporting success. An engine whose cells fail three times in a row is put down for the rest of the session and named in the failure file, because a thousand cells that each take their own time to fail is a day of the machine spent learning one thing. The next sweep tries it again, since the usual reason an engine failed is something somebody has since fixed.
• An estimate of how much longer a sweep has, from the runs it just did rather than from an estimate scaled off somebody else's silicon. It averages the last twenty, because a sweep walks from one thread up to sixteen and from pipeline one up to fifty and the cells are not the same size as each other, and it says nothing at all until there are a few, because an estimate from one run is a number with no information in it and people believe printed numbers.
• sweep --dry-run, which prints the cells it would measure in the order it would measure them and writes nothing, and sweep --cache, which sweeps one engine rather than seven. Between them they are how a sweep is checked before a machine is given up for a week.
• tools/provision/install.sh, which turns a fresh Ubuntu box into one that can run a sweep. It installs the build dependencies, a Rust toolchain, a .NET SDK for Garnet, memtier_benchmark and the seven cache servers, all at the versions pinned in tools/provision/versions.env, into the sibling directories config.jsonc already names. Every version it installs is written into every run file this harness produces, so the pins are part of the results rather than part of the setup, and two sweeps on two machines are comparable only if they agree. Running it again is the normal case: a checkout already there is fetched rather than cloned, and a binary already built at the pinned commit is left alone.
• Every crate is published to crates.io, so cargo install cache-bench is the answer to how this gets onto a machine that is about to be measured. The libraries go up with the binary because a binary cannot be published without the crates it is built from, and because the file model, the statistics and the chart engine are each worth reading on their own.
• A package job in CI that builds every crate from its own packaged tarball. A crate carries only what is inside its own directory, so an embedded file that reaches up out of the crate builds here and fails for everybody who installs it, and finding that out at tag time means a version on crates.io that cannot be replaced.
• The release workflow publishes the crates last, after every gate has passed and after the GitHub release exists, because a release can be deleted and a version on crates.io cannot. It skips a crate whose version is already up, so finishing a half published release is a matter of running it again, and it waits out the rate limit on crates that have never been published rather than failing, which the first release will spend half an hour doing.

Changed

• The golden fixtures moved from testdata/golden into crates/cb-core/golden, and every test in the workspace reads them through cb_core::golden rather than through a path that climbs out of its own crate. The fonts moved from assets/fonts into crates/cb-chart/assets/fonts for the same reason. This is what makes the crates publishable, and it is also what makes cache-bench verify carry its own evidence into the binary rather than depending on where it was built.
• The command line crate is called cache-bench rather than cb-cli, since that is the name of the binary it has always produced and the name somebody types to install it. The directory moved to match.
• config.jsonc looks for Garnet under net10.0 rather than net9.0. The pinned Garnet targets .NET 10 and pins that SDK in its own global.json, and the framework is part of the path its build writes to, so this line and GARNET_FRAMEWORK in the provisioner move together. The copy of the original's own config in the tests still says net9.0, because that file is theirs and quoting it wrongly would make the test prove nothing.

v0.5.0, the runner

This is the release where the project starts measuring. Everything before it was arithmetic over numbers somebody else took.

cache-bench run redis --threads 8 --pipeline 10 --perf no --run 1 measures one cell once and writes one file. Ask the binary its version, clear any leftover socket, start the server pinned to the cache half of the cores, wait for it to answer a real command over that socket, run a warmup pass and throw it away, attach perf if the cell wants counters, run the measured set pass and the measured get pass pinned to the other half, detach, stop the process group, confirm nothing survived, write. The sequence is the original's, in the original's order, and it does not vary by engine. Run it once per cell and the sweep is the loop around it, which is M6.

What is not the original's is what happens when the sequence goes wrong. Six things are refused here that the original would have measured through, and every one of them produces a plausible number rather than an error:

• A server left over from an earlier cell still answering on the socket. The original's answer is to pkill every process whose name it recognises, including ones it did not start, which on a shared machine kills somebody else's work and still leaves a full set of real numbers belonging to the wrong engine.
• Two sweeps sharing a results directory. That is not a race over a file, it is a race over the machine. Both pin the same cores and both bind the same socket, and the numbers come out low with nothing in them saying why.
• A perf cell on a machine whose counters do not answer, which would write a file full of missing counters that looks exactly like a machine where one counter is unsupported.
• More I/O threads than the profile pinned cores. Seventeen threads on sixteen cores measures the scheduler and draws a bar.
• A profile written for a bigger machine than the one running it. A pin naming some cores that exist and some that do not is narrowed by the kernel rather than refused.
• A memtier pass that did not complete the operations it was asked for, or came back missing a percentile.

A run that fails anywhere writes nothing, because a partial file cannot be told from a complete one by the stage that reads it next. A cell already on disk is skipped rather than measured again. That is the whole of how a sweep that takes days survives a reboot.

Also in this release: all seven adapters wired to one supervisor with no per engine special handling, readiness by protocol round trip rather than by sleeping, CPU pinning through sched_setaffinity between fork and exec rather than through taskset, a stopping sequence that confirms the process group is gone, the perf driver with counters taken over the server process only and only for the measured passes, a PMU probe that checks a live counter rather than the presence of a directory, and a strict memtier parser where the original uses a query that yields zero for a missing field.

The M5 exit condition is a gate and it has not been run. Seven servers each starting, answering, completing a run and stopping cleanly, with and without perf, needs a Linux host with all seven built on it. Everything up to that point is verified: unit tests on every piece, one whole run end to end against a cache server and a load generator that are not real, and thirteen checks on three operating systems. The gate is hardware work and it stays open until it runs on hardware.

The full list is in CHANGELOG.md.

v0.4.0

The documents.

Everything a reader sees around a chart is generated now. Both indexes, and a README in every results directory that says what the numbers were measured on, which versions were measured, what the method was and what the numbers may not be used for. None of it is written by hand, which is the point. 154 chart references maintained by a person go stale on the first rename and nobody finds out until a reader hits a broken image, and the original's README already disagrees with the data sitting next to it.

The generator is a function of its inputs and CI checks that on every push, twice over. It generates everything twice and diffs the two, and it runs docs --check over every published results directory so that a file somebody edited by hand fails the build rather than being quietly overwritten by the next sweep.

The exit gate was run rather than asserted. All 154 charts drawn, all three documents generated, then every image reference pulled out of the three documents and diffed against the directory listing. 154 references, 154 files, no diff in either direction. Deleting a chart produces documents that name it as not drawn and link to it from nowhere.

Still nothing measured. The runner is M5.

What is in it

• cache-bench docs, which writes LINEAR.md, LOGARITHMIC.md and the results README.md.
• results/<host>/host.json, the record of what a results directory was measured on, with nothing in it that names the machine.
• The divergences table and the caveat, generated into the README rather than kept in a document nobody following a chart link will open.
• A generated job in CI that catches both a non deterministic generator and a generated file somebody edited.

Four new divergences from the original, D16 through D19, all written up in divergences.md.

v0.3.0, the charts

The M3 mirror release. All 154 charts draw, and the same numbers produce the same bytes on Linux, macOS and Windows.

That last part is the milestone. The original's chart layer writes a matplotlib script, shells out to python3 and deletes the script, and what comes back depends on which fonts the machine happens to have, so two people running it produce two different pictures and neither can check the other. A chart drawn here is a function of its data and nothing else, and CI proves it on three operating systems on every push rather than asserting it in a README.

The renderer is written here, which was not the plan. Both ways plotters can draw text are closed to us: font-kit resolves system fonts, which is the thing being removed, and ab_glyph sits on a crate cargo-deny already fails the build on. What was left turned out to be small, because a chart is only two kinds of shape. Rectangles have an exact coverage per pixel with no sampling in it, and glyphs are outlines that skrifa reads out of the fonts already embedded here and zeno fills in scalar arithmetic with no SIMD path to diverge on.

The check that mattered most is the one that says the fixtures were not fooling us. All 154 charts drawn straight from the original's published output.json in upstream mode are byte for byte identical to the 154 drawn from the golden series committed here.

Two divergences came out of this release. Every chart is 1880 by 1130, where the original's own 154 come out in three widths because it crops each one to whatever it turned out to need. And every chart drawn from real measurements carries a line along the bottom naming the profile and the machine, because a throughput chart that does not say which box it came off invites a comparison that is not valid. Both are written down in divergences.md as D14 and D15.

Nothing has been measured yet. The runner is M5 and the first sweep is M6, so every chart drawn so far is drawn from the original's numbers.

v0.2.1

Everything a chart is, except the picture.

The chart engine is being built in two halves and this is the first one finished. What goes on each of the 154 charts and where it goes are both settled, both taken from the original rather than from reading the original, and both checked bit for bit in CI on a checkout with nothing measured in it. 154 titles, 11088 bars, 1872 ticks and 6098 gridlines, all of them the original's.

That was worth doing before the renderer rather than after. The original's chart layer leaves nothing behind but a PNG, so every decision it makes about what a chart says and where the axis starts is invisible once it has run. Both fixtures were taken by standing in for the thing it hands its work to, Python in one case and matplotlib in the other, which means neither is a transcription that could quietly drift from what the original does. Once the renderer exists, a wrong pixel will be a rendering bug and nothing else, because everything upstream of the pixel is already pinned.

The fonts are in the binary too, so nothing depends on what happens to be installed. Three faces, three licences, three digests checked against the embedded bytes.

M3 is not finished. The renderer, the hash manifest, the provenance stamp and the determinism job are what is left, and the exit gate needs two machines to agree on the bytes.

Added

• The series layer, which is the half of the chart engine with no pixels in it. A results file goes in and what comes out is a title, both axis labels, a thread count for each group of bars, and one number per bar, for each of the 154 charts. Everything a reader could disagree with is decided here, so it is a pure function and it is tested before anything is drawn.
• testdata/golden/series.json, which is all 154 charts as the original worked them out. Its graph tool pastes the numbers into a Python script and deletes the script after drawing, so tools/series-vectors stands in for Python, keeps the script and throws the picture away. The fixture is the original's answer rather than a description of it.
• Two levels of check on that fixture. The filenames, titles, axes, legend order and colours are checked in cargo test against a results file with the original's shape and none of its numbers, so it runs in CI on a checkout with nothing measured in it. The bar heights go through cache-bench verify --against, where they come out of our own reduction of the original's run files, which makes a matching chart one where every bar survived the run files, the selection, the combining and the extraction.
• The three faces the charts are drawn with, in the binary. Jost Book and Jost Bold for the original's Futura and its bold, DejaVu Sans for the Verdana it names on the quarter decade labels. Each licence sits in the directory next to the font it covers, assets/fonts/README.md records the exact release each file came from, and the SHA-256 of all three is written into font.rs where a test checks it against the bytes it embedded.
• The geometry layer. Where the y axis starts and stops, which ticks get a label, where the gridlines between them go, what number sits beside each one, how wide a bar is and where in its group it starts. All of it is the original's arithmetic, including the two different logarithms it uses for the two ends of a log axis and the eighth of a decade it calls a quarter.
• testdata/golden/axes.json, which is that geometry for all 154 charts as the original produced it. Its layout lives in about forty lines of Python inside two Go string constants and leaves nothing behind but a PNG, so tools/axis-vectors slices those constants out of cmd/graph/main.go and runs them with matplotlib replaced by something that records what it was told. Every bound, tick, gridline and label is compared bit for bit in cargo test, and cache-bench verify now prints an axes line making the same check from the command line. Like the series fixture it needs no measurements, because an axis is a function of the bars.

Fixed

• Kind now formats through pad, so a width in a format string does what it says. verify prints the four aggregates in a column and asks for eight characters, and the old implementation threw the width away without complaining, which is why that column was never a column.

Notes

• All 154 charts and all 11088 bars come back as the original's. That is the first half of the M3 gate met, and it is met before a single pixel has been drawn, which was the point of splitting the layer in two.
• Running the original's own layout code turned up three things that reading it did not. The thread count under each group of bars is placed at a hardcoded offset that is the middle of the group for six bars and for no other number, so our seventh engine would have knocked every x label off centre. That is D13 and the offset is now computed. The minor gridlines step by an eighth of a decade rather than the quarter their variable is named after, so there are seven between labelled ticks. And the only two charts with a zero on them are linear, which is the whole reason a zero has never reached the original's logarithmic path, where it has no answer.
• A bar can now be absent rather than zero. The original has no way to say that a cell was never measured or that the machine could not count cycles, so it says zero, and a zero bar claims an engine scored nothing rather than that it was not tested. On a logarithmic chart one of those takes the whole y axis with it. --compat=upstream still writes the zero.

v0.2.0

M2 is done, which means the statistics are finished and both modes are proved against the original's own data. Still nothing measures anything.

The milestone exits on a gate and the gate holds in one command. From the original's 20160 committed run files, upstream mode reproduces all 2304 of its chosen files byte for byte and the whole 1.7 MB of its published output.json byte for byte. Run cache-bench verify --against a checkout of the original and it will say so, in under a second, along with how far the corrected numbers sit from the ones it just reproduced.

That last part is the point of shipping both modes. The four defects now have sizes instead of descriptions. The typical median moves by a tenth of a percent on GET and a quarter of a percent on SET, and the worst median SET moves by 61 percent, which is Garnet at 8 threads and pipeline 50 published at 19.86 million operations per second where the median of its 31 runs is 12.30 million. The published median GET is the higher of the two in 576 of 576 cells, on every engine at every thread count and every pipeline depth, which is a chart that reads slightly fast everywhere rather than a chart with noise in it.

Worth saying plainly, because it is the argument for having built the gate at all: two of the four defects were described incorrectly when they were read off the Go source at the start of the milestone, and byte parity is what corrected them. Two more behaviours were not visible in the source at all. The info block of a chosen file comes from the last run read rather than from the run selected, and cleanperf rewrites exactly six counters and leaves the others alone. Neither is guessable and both are needed.

M3 is the charts, which is where the numbers finally become something to look at.

Added

• cache-bench choose, which reduces every cell in a results directory to its median, best, worst and average. --compat=upstream reproduces the original's four defects, --out writes somewhere else so that two modes can be compared without either overwriting the other, and --cell does one cell for when you are looking at a single number rather than a sweep.
• cache-bench combine, which gathers the chosen files into the output.json the charts read. No computation in it. Every number was decided by choose and this collects them in the order a directory listing gives them, which is the original's order because the original builds the file straight out of one.
• cache-bench verify, which is the claim this port makes about itself run as a command. With no arguments it checks the golden files committed here and runs anywhere in under a second, which is why it is in CI. Pointed at a checkout of the original with --against it reads all 20160 committed run files, reproduces all 2304 chosen files and the whole published output.json byte for byte, and then prints how far the corrected statistics sit from the original's. The numbers in divergences.md under D1 to D4 are that output rather than an assertion about it.
• The results directory layer the two of them share. A gap in a cell's run numbering stops that cell at the gap and says how many files sit above it, rather than reducing 30 runs and calling them 31.

Changed

• The eight crates in the workspace are marked as not published, and the version requirements on the paths between them are gone. Those requirements have to move in lockstep with the workspace version or the build stops resolving, which is what happened when this release was first cut, and they buy nothing when the crate is never resolved from a registry. Nothing here goes to crates.io and the manifests now say so.

Notes

• The M2 gate is met end to end. From the original's 20160 committed run files, choose --compat=upstream writes all 2304 of its chosen files byte for byte and combine writes its published output.json byte for byte, all 1.7 MB of it.
• That makes the size of the four defects a measurement. The same directory reduced in corrected mode moves the typical median by a tenth of a percent on GET and a quarter of a percent on SET, and the worst median SET by 61 percent, which is Garnet at 8 threads and pipeline 50 published at 19.86 million operations per second where the median of its 31 runs is 12.30 million. The published median GET is higher than the true median in 576 of 576 cells, which is what a one sided index error looks like once you can see all of them at once.