v0.3.30

Eight pull requests and no milestone has closed, so this is a patch.

The headline is that this build is a master now. A real redis-server 8.10.1 told to replicaof yo attaches, takes a snapshot, follows the command stream and ends up holding the same keyspace, and it stays that way across a broken link because partial resync works too. Two more of the metadata commands land beside it, being the rest of the ACL and the whole of MEMORY, which is the last of M8's looking commands. The other four are keeping the tree buildable and measurable, which is one real throughput fix on many threaded servers, one more pass at the Miri run, one release job that gave up too easily and one Windows test that was asserting a sentence from another operating system. Nothing here changes the on-disk format and no record kind was added.

Added

• Replication, the master half. REPLCONF, PSYNC and SYNC are new commands, and PSYNC is where a connection stops being a client: it gets +FULLRESYNC <replid> <offset>, then the snapshot as a bulk string with no newline after it, and from then on the socket carries commands and is never replied to again. Everything that connection was holding as a client goes at that moment, being an open transaction, the keys it was watching and any subscription, because all three are promises of a reply and a stream has no way left to keep one. Answering a REPLCONF ACK with an +OK puts a reply into the middle of a byte stream the replica is parsing as commands, which a real replica reports as Protocol error (Master using the inline protocol. Desync?) before it drops the link, so the silence there is the protocol rather than an optimisation. What crosses the link is the effect and not the command, which is the reference's own rewrite table followed case by case: a relative EX crosses as the absolute PXAT it resolved to, SPOP crosses as the SREM it turned out to be, INCRBYFLOAT crosses as a SET of the answer, a key a read noticed had expired crosses as a DEL ahead of whatever the read was doing, and a script crosses as each inner write on its own rather than as the EVAL. SELECT is injected lazily, exactly when the database a command is on is not the one the stream is already pointed at. WAIT counts replicas that have acknowledged rather than answering zero.
• MEMORY USAGE, STATS, DOCTOR, PURGE, MALLOC-STATS and HELP, which is the last of the metadata commands on M8 and closes the last four COMMAND GETKEYS differences. USAGE adds up the record in the arena, the body hanging off it for a type that has one, and one key's share of the index, which is the same three things a real server adds up under different names, so the shape of the answer matches even though none of the three numbers does. It is exact rather than sampled, because every collection already keeps its own byte count for maxmemory, so SAMPLES is parsed and checked and then changes nothing, which is D-6. A value that has been written out to the file is counted as the record alone, because that is all of it that is in memory, and it is deliberately not read back in to answer, since asking what a key costs should not be the thing that makes it cost that. STATS answers the same 36 fields in the same order plus a row per database that is holding a key, and the fields that are zero are zero either because nothing is behind them yet or because of a real design property, there being no expires table when a deadline lives in the record and no half rehashed index. DOCTOR reports five of the reference's eight complaints, in the reference's own wording, and leaves out the three that need a resident set size or a replica rather than shipping a complaint that can never fire.
• ACL LOG, ACL LOAD and ACL SAVE, which is the rest of the access control list. LOG is a ring buffer of refusals, newest first, at most acllog-max-len of them, written by every refusal the gate makes and by a failed AUTH with the name that was tried rather than the name the connection is on. Two refusals merge into one row when the reason, the context, the object and the user all match and they are within sixty seconds of each other, and the scan that looks for a match reads at most the first ten rows, which is the reference's rule and is why an eleventh distinct refusal can stop a match being found. The log lives on the server with its own lock rather than inside the user table, because the two are touched at opposite moments, so a server whose users can do what they ask never takes the log's lock at all. SAVE writes one user a line in name order to a temporary file beside the real one, fsyncs it, renames it and fsyncs the directory, which is what makes a half written ACL file impossible, and LOAD stages every rule off to the side so a file with a mistake anywhere in it changes nothing at all and every complaint it has comes back in one sentence. yodb takes the file as --aclfile, and a server that was not given one answers both with the reference's paragraph about ACL SETUSER and CONFIG REWRITE.

Changed

• Each thread gets its own compaction cursor. Chasing the maintenance slice's nine per cent on issue #334, the paired attribution on the 32 thread box said DEBUG PAUSE-CRON 1 is worth 1.09x at 16 threads while neither of the two jobs inside the slice that have their own knob is worth anything measurable, which is what sent me to read the slice rather than the jobs. Three of its four steps cost one relaxed load on a default server. The fourth stored Server::next_db on every batch whether or not the walk found anything to move, because the store is how the next walk is told where to start, and a walk that finds nothing is nearly every walk on a server that is keeping up. So it was sixteen writers on one cache line at batch rate, with a cost that goes up with the thread count rather than staying still, which is the shape the attribution table has.
• The last four yo-kv tests over three minutes interpreted now run in under twenty seconds each, which is the fourth pass at the bar issue #499 set and the same rule as the three before it. A count that is the claim of a test gets skipped under Miri with a reason naming the count, and a count that is only a way of reaching a state gets smaller with the ratios held. The one worth looking for first is the fourth, at 204s to 5s with nothing cut at all: it was slow because the helper building its test bytes ran a modulo per byte over about eight hundred thousand bytes, and it now builds one cycle of two hundred and fifty one and repeats it, so the bytes and every size the test walks are unchanged.

Fixed

• Partial resync never happened, so the backlog was doing nothing at all. The offset a replica sends with PSYNC is the position of the first byte it wants counted from one, so a replica that has everything asks for one past the end of the stream, and everything on this side counts bytes written. Reading one as the other put every request past the end of the backlog, and the replica log said so plainly once you look for it, logging Trying a partial resynchronization (request <id>:128272) and getting Full resync from master: <id>:128271 back. repl_backlog_first_byte_offset in INFO was off by the same step and is now reported the way a replica counts, being zero when there is no backlog rather than the one that would be the first byte of the one that was never made.
• CLIENT KILL TYPE replica returned zero and connected_clients counted the replica. The client type filter was written when there were no replication links, so the word matched nothing. A replica is now its own type, normal and pubsub exclude it the way getClientType does by asking its questions in order and stopping at the first yes, and connected_clients is sockets minus replicas.
• A release that could not reach a registry gave up on the first try. cargo xtask reserve verify is the gate the release runs before it publishes anything and it exits 2 when it could not get an answer rather than pretending the name is fine, which is the right call and stays. On 8 September one read against search.maven.org timed out and the v0.3.28 release stopped there, so 0.3.28 has a tag and no crates.io release and no GitHub release, and nobody noticed for a day because the failure was in a job there is normally no reason to read. It now asks three times, two seconds then four between them, and only for a request that did not come back at all, because an HTTP status is a verdict including a 404 and a 403 and asking again would get the same one.
• A test asserted that errno 2 reads No such file or directory, which it does out of a C library and does not out of Windows, where it reads The system cannot find the file specified., so test (windows-latest) went red on main. The function under test is fine and portable, since all it does is take off the (os error 2) that Rust adds to the system's own sentence, so the test now asserts the two things that are true everywhere and keeps the exact C sentence under cfg(unix), which is the platform the reference runs on and where matching it word for word is the point.