v0.3.33

Five pull requests between milestones, so this is a patch. Four of them finish atomic slot migration and add the election a cluster needs to survive a node going away, and one takes an allocation, an index insert and an index remove off one write in seven. Nothing here changes the on-disk format and no record kind was added.

Added

• A slot migration finishes when the far side claims the slots. Everything up to this was already in, the task, the snapshot, the change stream, the write pause and the STREAM-EOF that says nothing more is coming, and what happened next was nothing: this node heard the claim, updated its map, and left the migration sitting in stream-eof until the write pause ran out and gave up on it. The bus now carries the slots this node lost out of the map lock and hands them to the migration, and a task moving exactly those slots and sitting in stream-eof is a migration finishing, so it completes, the pause lifts and the keys behind the slots are dropped. Exactly those slots is the reference's rule and it is stricter than it looks: a claim covering half of what a migration is moving is something else happening to the cluster while a migration ran, and it fails the migration rather than reporting it done. How the drop reaches a follower has three shapes, and they are three situations rather than a choice. A migration finishing writes one TRIMSLOTS naming the ranges and fires a del event per key, because an operator asked for this and is watching. A slot that changed hands by failover or by SETSLOT sends a deletion per key and fires nothing, because nobody asked for those keys to go. A TRIMSLOTS arriving from a master fires the events and propagates nothing, because the command that asked for it is already on the stream.
• CLUSTER MIGRATION IMPORT, so a node can take slots as well as give them up. This was the last piece of the protocol missing, and it is what stopped the shipped resharding tool working against a cluster of these at all: redis-cli 8.10 picks the new way whenever every node reports 8.4.0 or above, and the first thing it does is tell the destination to import. A node told which slots it wants works out who owns them, refuses the five ways the reference refuses in the reference's order, books a task, answers with the task id straight away, and goes and gets the slots on a thread of its own. It opens the same two connections the reference opens, applies the snapshot while the changes pile up unread behind it, works through the pile, follows the stream, and claims the slots when the source says the stream has ended. The count it acknowledges is decoded command bytes with the control commands left out, which is the same rule on both sides, because the source stops taking writes the moment what it has been told matches what it has sent. Two real bugs came out of writing it. The source held the snapshot connection after the snapshot had been read, so a destination closing its end, which this and a real redis destination both do the moment the last of it lands, read as a channel dropping under a live migration and failed the whole move. And an import that failed partway left the keys it had applied sitting in slots this node does not own, so it now trims on the way out as well as on the way in, and only the slots that are still somebody else's.
• A replica stands for election when its master fails, and a master here votes in one. This is what decides whether a cluster survives a node going away without somebody watching it. The election is the reference's, check for check: half a second plus a random half second plus a second for every replica of the same master holding more data, then bump the epoch and ask every node. A master answers if it serves a slot, if it has not already voted this epoch, if the node asking really is a replica of a master this node agrees is failed, if it has not voted about that master in the last two node timeouts, and if the slots being claimed have not moved under a newer epoch. There is no no vote on this protocol, so a master that disapproves and a master that has crashed look the same from where the replica stands, which is deliberate, because the alternative is a reply a replica could be made to wait for. The epoch a node last voted in now goes in the config file where the reference puts it, and it was the one field there being written as a constant nought. One real bug came out of running it on live nodes: failure detection measures the time since a ping went out, and a node with no link never gets one sent, so a node whose address stopped answering was dialled again every tenth of a second for ever and never marked failing. The first run sat at fail question mark for sixty five seconds and got no further. A real server has the same problem and solves it in the same place, by saying it sent a ping when the connect fails and by keeping an outstanding ping across a reconnect rather than restarting the clock, and both are in.

Changed

• An overwrite stays where it is when the new value fits in the run. RawMap::set_with wrote a record over itself only when the new value was exactly the length of the old one, and allocated a new record and marked the old one dead for every other length. Exactly, because nothing in a record said how much room it occupied, so the two lengths in its header were the only thing a walk could step by, and a short value in a long run would have sent that walk into the middle of the next record. The room a record has and is not using is written down now, in the top eight bits of the second header word, in units of the arena's sixteen byte alignment, and the header is still eight bytes: a record cannot be longer than a segment, so a value length has never needed more than twenty one bits and those bits have always been zero. So a record occupies exactly what its header says and a walk steps by the run. An overwrite stays in place when the new record fits and would leave no more than a quarter of the run behind, which bounds what a key can hold unused rather than letting it settle at the largest value it has ever held, and compaction is where the room goes back, since it is the one moment a record is rewritten with nobody waiting on the reply. Measured over twenty thousand keys with values drawn from 1 to 1024, five passes, which is the shape the cache benchmark runs: 14.5 percent of overwrites stay in place for exactly the same number of arena bytes as before. Half the run would keep 30 percent and cost seventeen percent of the arena, and the whole run would keep 68.8 percent and cost two thirds of it. MEMORY USAGE counts the run now rather than the header and the name and the value added up, which is how a real server answers it, by asking the allocator how big the block really is.
• CLUSTER SETSLOT is refused for a slot inside a live migration. There are two ways to move a slot and they write the same three fields from two directions, so using both on one slot ends with the migration claiming a slot the operator had already handed to somebody else. A real server refuses the old way and names the cancel that gets you out of it, and that sentence is worth copying exactly because a resharding tool is what reads it. Checked against a real 8.10.1 with a task running on both servers in the same state, five cases and no difference. The reference has a second gate there for a slot with a trim job outstanding, which cannot happen here because a trim runs on the spot rather than being queued.
• The drain timeout is read rather than ignored. It was the last of the four migration settings that was in the config table and read by nothing, and it catches a move that is not broken but never ends: the far side gets through everything behind the snapshot and says so, then stays further behind the stream than a handoff may start at, because this node takes writes faster than that side applies them. Nothing is wrong with either end and left alone it runs until somebody notices. The deadline is the longer of the setting and twice however long the far side took over the snapshot and the pile, since a side that needed a minute for the first part is not one to give ten seconds for the rest, and the span starts where the backlog ended rather than where the move did, so a far side still working through the pile is never given up on for taking a while. With eight writers pushing four kilobyte values faster than the destination could apply them, the task sat in send-stream for twelve seconds with the setting at one millisecond and was not touched, and completed on its own the moment the writers stopped with all 23831 keys across.

Performance

• The eight core box measured yo at v0.3.31 over the same twelve cell matrix the published epyc8coarse directory holds v0.3.28 in, five runs a cell, and this is where the threading milestone's first gate stands rather than a claim about this release. Sets at depth went up by two to three and a half times: 132 to 457 thousand a second at pipeline 50 with four threads, 140 to 418 at pipeline 25. Gets at one thread went down by a fifth to a quarter at pipeline 10 and deeper, which is open as #541. Against the best of the seven rivals in the same cell the best yo reads is 0.97x and the worst is 0.43x, against a gate of 2.00x, so the milestone has built what the gate needs and has not come near it. These are development measurements at five runs a cell where twenty one of the twenty four cells vary by more than a twentieth, so they are ratios and not rates.
• Memory, on the same box, eight engines holding 2,499,968 keys with values drawn from 1 to 1024, at peak resident: yo pays 45.0 bytes an entry over the payload, against rugo at 24.7, dragonfly at 41.0, pogocache at 89.8, valkey at 103.4, redis at 116.2, memcache at 146.6 and garnet at 911.3. The payload is 518.6 bytes an entry, so total resident memory is 1.41 GB against rugo's 1.36, and the overhead is the column with anything in it. Development measurement, one pass, not a gate number.

Known gaps

• Gets at one thread lost a fifth of its rate at pipeline 10 and deeper between 0.3.28 and 0.3.31, and one thread is the case where none of the threading machinery is being used. It is open as #541 and it is the only regression on the threading milestone that is not a tradeoff for something. The serve bench cannot resolve it on the eight core box, because the clients are on the same cores as the server and read a coefficient of variation of 0.35 for a cell the harness resolves in three hours.
• What the in place overwrite is worth in throughput has not been measured on a box where the load generator is pinned away from the server. Paired on an arm laptop it reads 1.03x with fifteen of twenty one pairs on the same side of even, which is what a machine with the clients fighting the server has to say about a one percent effect.
• Cycles per operation has no measurement on either box. epyc8coarse sets perf = ["no"] on purpose, because the full epyc8 matrix with both halves of that flag is eight thousand cells and weeks of machine time, so a number for the milestone comes from perf against one cell rather than from a sweep. Nothing is broken and the box has a live PMU.
• Manual failover is still not in. CLUSTER FAILOVER here refuses, and a yo master asked to hold its clients still does not, so a real replica asking waits its five seconds out. The flag that says vote for me even though my master is up is honoured, so a real server's replica doing a manual failover in a mixed cluster does get a vote from a yo node.
• 0.3.28 has a tag, no crates.io release and no GitHub release, from a probe that timed out on 8 September. It stays a gap rather than being published out of order behind the four releases after it.