v0.3.32

Six pull requests between milestones, so this is a patch. Four of them are the next steps of a slot migration, one takes a per batch cost off the command path, and one stops the server lying about how many threads it has. Nothing here changes the on-disk format and no record kind was added.

Added

• A slot range is sent as a snapshot. The node giving slots up now does its half of an atomic migration. A node that has logged in as the cluster asks with CLUSTER SYNCSLOTS SYNC, is told to open a second connection, and when it opens that one with CLUSTER SYNCSLOTS RDBCHANNEL it is sent a snapshot of exactly those slots. The snapshot is a stream of ordinary commands rather than an RDB file, which is the only way a sixteen thousandth of the keyspace can be sent at all: the libraries, a SELECT for every database that holds anything, a count of the keys and the deadlines of each slot in front of the first key of it, the keys themselves, and an end marker. A string goes as the SET that would set it, which is what the reference does so the far side can take a large value apart as it arrives, and everything else goes as a RESTORE with an absolute deadline, because the far side runs these at some unknown moment after they were written and a duration would restart the clock. It is taken with every write held off, which is what this has instead of a fork, and that freeze has been pulled out of the replication code as at_an_instant so the full resync and the migration use one piece of machinery. CLUSTER MIGRATION STATUS and CANCEL now report and cancel the real task rather than answering empty and nought.

• The four slot migration settings are in the config table. cluster-slot-migration-handoff-max-lag-bytes at 1048576, cluster-slot-migration-write-pause-timeout at 10000, cluster-slot-migration-sync-buffer-drain-timeout at 60000 and cluster-slot-migration-max-archived-tasks at 32, with the reference's defaults, units, ranges and refusals, and the last two hidden the way the reference hides them, which means a pattern does not find them and their own name does. The lag bound and the archived count were hardcoded and are read off the settings now. The first counts bytes and takes a unit, so 2mb reads back as 2097152 the way maxmemory does, and the other three count something else and refuse 10s with the reference's sentence about not being able to parse an integer.

• The writes that land behind the snapshot are streamed after it. Every command that propagates goes past the migration on its way to the replicas, already rendered in the form a replica would see, and is kept or dropped on the slot its key is in. No keys, dropped. Keys outside the moving ranges, dropped. Keys in two different slots, the migration is cancelled, which is what a real server does with it too. The stream is switched on inside the same freeze the snapshot is read under, so the two meet exactly and nothing is sent twice or missed. CLUSTER SYNCSLOTS ACK now records how far the destination has got, refuses to go backwards, and moves the task to handoff-prep once the destination is within a megabyte of what has been sent.

• The writes stop and the stream ends when a moving slot changes hands. Once the destination is within cluster-slot-migration-handoff-max-lag-bytes of everything that has been sent there is nothing left to catch up on, so the source stops taking writes, waits out the ones already running, sends CLUSTER SYNCSLOTS STREAM-EOF and waits for the far side to claim the slots. Reads keep working the whole time and a write is held rather than refused, so a client writing during a handoff sees a pause and not an error. The reference does this by waiting for a socket buffer to drain, which it can because it is single threaded and nothing else runs in between. Here the writes are on other threads, so the pause goes on first and then the same freeze the snapshot is taken under is used as a barrier: it touches every stripe, and a write feeds the migration stream while it still holds its stripe, so coming out the other side means every write in flight has already been sent. The pause carries its own deadline as well as being lifted by hand, so a bug in the lifting cannot leave a server refusing writes for good, and a destination that never takes the slots is given up on after cluster-slot-migration-write-pause-timeout with the slots and the keys where they were.

Changed

• A memory reading is taken once a millisecond and weighs only the databases that moved. On a server with --maxmemory set, the maintenance turn after every batch folded a walk over every stripe of all sixteen databases, locking each one and asking it for its slab bytes plus seven collection totals. Fifteen of those sixteen are empty on every benchmark cell there is. A batch is a hundred nanoseconds, so that was ten thousand readings a millisecond of a number that moves by what sixty four commands allocated. The reading is now gated to one a millisecond per thread, and it re-reads a database only when something has marked it since, keeping each database's last figure beside the running total. What a reading that old costs is overshoot bounded by what a millisecond of writes can allocate, which is well inside the tolerance this number already had, because space comes back a segment at a time and a segment is two megabytes. The server sitting at its limit is not judged on this reading at all: make_room takes its own exact one the moment the cached figure says the server is over.

Fixed

• io_threads_active and io-threads report the real thread count. Both were written down rather than read. INFO server had io_threads_active:0 as a literal inside the format string and io-threads sat in the fixed settings table with the value 1, so a server started with --threads 4 told every client it had one thread and was not using it, while the # Threads section of the same reply reported four. All three places now read the count off the server. Redis means the configured count by io_threads_active rather than the number of threads with work in hand, and matching Redis is the reason the field exists, so that is what this matches. CONFIG SET io-threads keeps the rule the fixed settings follow: a write of the count the server already has is taken and everything else is refused as immutable.
• A failed migration named the wrong state and the RDB channel always said none. A failure sentence named the state the task had just moved to where the reference names the state it was in when it went wrong, because the reference formats the message before the caller moves the state, and rdb_channel_state said none where the reference says completed once the snapshot has gone out.
• CONFIG GET matched a name with regard to case. CONFIG GET MAXMEMORY answered nothing at all, which is the sort of thing a client library written against a real server trips over on its first connection. A real server hashes the exact name through a case insensitive dictionary and runs a pattern through its matcher with the nocase flag set, and that is what this does now. The name a setting is answered under is the one the client spelled when they spelled it out and its own name when they gave a pattern, so CONFIG GET MAXMEMORY answers MAXMEMORY and CONFIG GET MAX* answers maxmemory, which reads like a quirk and is really the shape of the reference's code.
• SPOP with one member left crossed to a replica as DEL. A real server always sends SREM with the member it took and lets the far side notice it is holding an empty set. Only the form with a count sends DEL, and only when the count takes the whole set. This was wrong for ordinary replication and not only for a migration, and the new migration differential is what turned it up.

Known gaps

• What the memory reading change is worth has not been measured. Both benchmark boxes are running the yo only sweeps the threading milestone needs and will be for hours, so a before and after against the same harness has to wait. Six percent is the ceiling it can recover, from the same cell reading 133578 sets a second with the limit off against 125214 with it on, on the eight core box at pipeline 50.
• An overwrite only writes in place when the new value is exactly as long as the old one, because nothing in a record records how much room it occupies. A fixed length cell runs at about 550 thousand sets a second on the eight core box and the same cell at --data-size-range 1-1024 runs at about 250 thousand. That is the other half of the write path gap and it is open as #531.
• A slot migration still has no last step: the far side claiming the slots over the bus and this node dropping the keys it no longer owns. There is no CLUSTER MIGRATION IMPORT either, which is a node asking to take slots rather than being asked to give them up.
• 0.3.28 has a tag, no crates.io release and no GitHub release, from a probe that timed out on 8 September. It stays a gap rather than being published out of order behind 0.3.30 and 0.3.31.