v0.3.29

Eight pull requests and no milestone has closed, so this is a patch.

Two of them are command surface and they are one piece of work in two halves: the key specs Redis added in 7.0, and then the access control list that could not be built without them. That is the biggest part of M8 that was still missing, and a server that had one password and one user holding it now has as many users as an operator cares to write. The other six are about being able to build, measure and test this thing at all. Three are the throughput bench learning to refuse a machine it cannot trust, which matters because every box available this week has somebody else on it, two are the nightly Miri run, which had stopped finishing, and one is a link error on Windows. Nothing here changes the on-disk format and no record kind was added.

Added

• Key specs on every command row, COMMAND GETKEYSANDFLAGS, and a key finder the ACL can use. The legacy first, last and step triple was never enough to say where a command's keys are. ZUNIONSTORE dst 2 a b has three keys and the triple names one, XREAD COUNT 2 STREAMS a b 0 0 has two and the triple names none, SORT k STORE d has two and the triple names one. That was survivable while the only caller was COMMAND GETKEYS, and it stops being survivable at the ACL, because a user given ~cache:* must not reach secret through ZUNIONSTORE, and a permission check built on a key finder that misses keys is not a permission check. The resolver is getKeysUsingKeySpecs line for line, including the three rules that read as mistakes and are not: a run reaching past the last argument is a syntax error rather than a shorter run, a keyword search stops one argument short of the end because a keyword in the last argument has no key behind it, and a spec that says it is incomplete throws away the whole answer rather than half of it. The specs are tried first and a per command finder is what is left when they fail, which is the order getKeysFromCommandWithSpecs goes in, and PFMERGE dst is the case that proves the order matters, since its source run starts at argument two of a two argument command.
• Users, the ACL rule language and a gate in front of every command. A user is a name, a switch, a list of SHA-256 password hashes and one or more selectors, and a selector is a command bitmap with a list of key patterns and a list of channel patterns beside it. That is the reference's model down to the reserved bit that records whether a command added tomorrow would be allowed, which is what makes +@all -get and -@all followed by the other four hundred names two different users rather than one: they describe back differently and they behave differently the day a command is added. Every selector also keeps its rules as the text they were written in, because ACL LIST and ACL GETUSER have to hand back something SETUSER will take again, and a bitmap cannot say whether what was written was a category or the forty names inside it. The rule language is ACLSetUser and ACLSetSelector written out, refusals included, each wrapped in the sentence acl.c wraps them in. Ten subcommands ship, being SETUSER, GETUSER, DELUSER, LIST, USERS, WHOAMI, CAT, GENPASS, DRYRUN and HELP, and ACL HELP does not name the three that are still to come, because a client reading the help to find out what it can send should not be told about a subcommand this server would refuse as unknown. A SETUSER is staged on a copy and applied only if every rule worked, which is the reference and which matters because an operator tightening a user and mistyping one rule would otherwise be left with the new restrictions and none of the new grants. The gate sits where processCommand puts it, after the refusal for a command that cannot go inside a transaction and before the memory limit, and on a server whose users can all do everything it costs one relaxed load. requirepass is now one rule on the default user rather than a password of its own, and ACL GENPASS takes its bytes from /dev/urandom or BCryptGenRandom rather than from the engine's seeded generator, which closes D-127.

Changed

• The yo-cli throughput bench now refuses a machine it cannot trust. A ten core laptop with a virtual machine, two builds and a Python job on it answered 3220 Kops at one thread where the same build on the same laptop had answered 6417 an hour earlier, and nothing in the output said so. A number like that does not look wrong, it looks like a regression, and acting on it is worse than having no bench at all. Every cell is measured three times, what gets reported is the median and the coefficient of variation across the three, and a run with cells over five percent says in as many words that nothing in it is worth quoting. The median rather than the mean, because a run that lost the machine to something else is an outlier rather than evidence, and the spread beside it because the whole point is that it is reported instead of being averaged away.
• YO_BENCH_AGAINST turns that bench into an A/B against a second yodb binary, which is what makes it answer on a busy box instead of only refusing one. Three machines in a row had refused it. Each repeat measures the cell twice, once with each binary, back to back and in alternating order, and the ratio is what gets reported. The thing that makes it work is not the pairing, it is the median over enough pairs: a pair cancels slow drift and does not cancel a compiler taking half the cores for six seconds of one half of it. Measured with the binary against a copy of itself on a laptop at a load average of ten, where the true answer is 1.00x, three pairs said 1.16x and eleven pairs said 1.02x, while the absolute throughput moved by a factor of two between the two runs. A cell is judged on how many of its pairs agree rather than on the spread.
• YO_BENCH_DEBUG_AGAINST sets a DEBUG knob on one side of a pair only. The attribution table that says what each background job costs was four separate runs of the same build with a different job turned off in each, which is why it was taken on a quiet machine and why it could not be taken again on a busy one. Worse, YO_BENCH_DEBUG reached both halves of a pair, so a paired attribution compared a build against itself with the same job turned off on both sides and read 1.00x by construction. Unset, the far side is set up the same way the near side is, which is what a comparison of two builds wants and is what happened before. Set, including set to nothing, the two sides differ by that and by nothing else, and both settings are printed in the header so a number pasted into an issue carries what it was measured with.

Fixed

• The nightly deep run failed on three days running, for two reasons that were not bugs in the server. Net::held in yo-cli is cfg(all(test, unix)) and its only caller is cfg(all(test, not(miri))), so under Miri the caller went, the method stayed, and a shard built with -D warnings could not get as far as listing its tests. And one yo-resp test runs an EVAL, which needs a Lua state, which is C. Every other test in the tree that touches Lua already carries the same ignore attribute with the same words, and this one was written without it.
• The yo-kv Miri shards ran past their forty minute timeout. Interpreted, a member of a sorted set costs somewhere near a second to build against about fifteen milliseconds for a member of a plain set, and that gap is why counting less was not enough on its own: a sorted set test needs a count past the listpack band or it is testing the packed case instead, and a hundred and twenty nine members at a second each is over two minutes before the test does anything. So where a test crosses a band rather than counts to a number, the band comes down under Miri and the counts either side of it come down with it, which leaves the same boundary in the same code with a twelfth of the members to carry over it. Three tests over a demoted body went from over five minutes each to under three seconds that way. A third pass timed the tests that reading the source had missed, because a bound written as 4_000usize has no word break in it and a bound that is a band is not a large number at all, and found ten more between a hundred and six hundred seconds which now run in six to twenty one seconds. Every one of the eight shards now finishes between eight and twenty one minutes. The rule behind it is in deep.yml: a count that is the claim of a test does not get smaller, it gets skipped with a reason naming what the count is for, and a count that is only a way of reaching a state does get smaller, with the assert that would go vacuous checked by forcing the Miri branches on and running the suite compiled.
• The Windows build stopped linking the moment BCryptGenRandom got a caller. The library it lives in has to be named, and nothing else in the tree pulls it in. The standard library used to, back when its own generator was this same call, and it has since moved to ProcessPrng in another library, so a call that linked by accident stopped linking when the toolchain caught up. One attribute fixes it. It is worth writing down because of how it fails: the error is at link time, only on the MSVC target, which is not a target a person developing this is usually on, so the first thing that saw it was CI on the release branch.