v1.2.0

KiviDB v1.2.0

What's in 1.2.0

1.2.0 is mostly about not losing writes, and being a better Redis:

• Durability.
  • With the append-only file on, a write is now acknowledged only once its record is in the file, as in Redis, so a killed process keeps every write it acknowledged, as long as the disk keeps up: a reply waits at most 5 s for its record, and one sent without it is counted in INFO persistence as aof_acks_before_write.
  • A checkpoint no longer stalls writes for seconds on a slow disk.
  • A graceful shutdown (SHUTDOWN, SIGTERM) keeps every write it acknowledged: no command runs from the start of the final snapshot to the exit.
  • A transaction, or a script that writes more than once, reaches replicas and the AOF as one step, so a replica never shows it half done.
  • A manual failover whose election is slow no longer loses writes.
  • Automatic failover elects the replica with the most data, and a replica stands only once it holds its master's data: never in the middle of a full sync, and its data's age counts from when it was last in step.
  • A clean shutdown waits up to shutdown-timeout (10 s) for its replicas to have every write, so after a restart they carry on instead of being sent the whole dataset. A manual failover does the same for the master it replaces.
  • A full sync under heavy writes finishes: while the replica loads the dataset, the writes kept for it may grow to repl-replica-sync-max-backlog-bytes (1 GiB) instead of the normal limit, and both limits can be changed with CONFIG SET.
  • A replica stopped in the middle of a full sync keeps its last good dump.kdb instead of saving the partial dataset over it.
  • A server started in a data directory another one is still using, as a replacement pod can be after a forced delete, waits for it to exit instead of loading what it was about to replace.
• Clusters.
  • The bus keeps one connection per peer.
  • Nodes can announce a separate client port (cluster-announce-client-port), for NodePorts and load balancers.
  • After a failover, the shard's other replicas carry on instead of resyncing in full.
  • An import cut short by a restart cleans up after itself.
  • redis-cli --cluster reshard runs to the end under writes and loses none of them: writes to the keys a MIGRATE is moving wait for it, as in Redis. redis-cli --cluster fix closes a reshard that stopped halfway, and no key is left on the wrong node: CLUSTER DELSLOTS works as in Redis, and DEBUG DIGEST-VALUE returns each value's digest.
• TLS without restarts.
  • Certificates, CAs and tls-port change with CONFIG SET.
  • A server can turn TLS on while it runs.
  • port 0 runs a server on TLS only.
  • tls-auth-clients-allowed-names accepts only client certificates whose names match.
• Memory.
  • Active defrag gives memory freed by deletes back to the system, and is on by default.
  • String values take less memory.
• Multi-key writes. MSET and MSETNX of keys in many shards no longer queue behind each other: more than three times as many a second as 1.1.3 with 1,200 clients, and a lower tail.
• Search. TEXT fields are stemmed, as in Redis: preferences finds prefer. Scores come out as Redis's, and SCORER is honoured. FT.SEARCH and FT.AGGREGATE take a TIMEOUT, and the timeout is safe across a rolling upgrade. Synonyms work. Scores stay right after updates and deletes.
• Compatibility. Dozens of fixes bring KiviDB's replies, errors, INFO and CLIENT LIST fields, keyspace events, Lua scripting and vector-set scores closer to Redis 8.10.
• Signed releases. Every file and image carries a signed build attestation. Check one with gh attestation verify <file> --repo kividbio/kividb.

CHANGELOG.md has the full list.

Security

1.2.0 fixes issues that let a user restricted by ACL read or overwrite keys outside their permissions, or read credentials. 1.1.x is affected too and gets no further release, so upgrade to 1.2.0. Thanks to @Parikalp-Bhardwaj, who reported them privately.

• ACL key permissions bypassed through SORT patterns and multi-STORE destinations (GHSA-6rw7-4fcr-r6j3):

  • SORT patterns. A user whose ACL names some keys could read any key's value, or any hash field, with SORT/SORT_RO ... GET * or GET *->field, and learn the order of other keys with BY *. Such patterns now need read access to every key, as in Redis.
  • A second STORE. SORT with two STOREs, and GEORADIUS/GEORADIUSBYMEMBER with two STORE/STOREDISTs, wrote to a key the ACL had not checked.
  • XREADGROUP. With a group or consumer named STREAMS, it read streams the ACL had not checked.

• Credentials in SLOWLOG, MONITOR and the log (GHSA-c4xj-63wm-4jj9):

  • SLOWLOG GET. A user allowed it could read the passwords set with ACL SETUSER, and the values of requirepass, masterauth, primaryauth, cluster-bus-secret and the TLS key passphrases set with CONFIG SET.
  • MONITOR. It showed MIGRATE's AUTH/AUTH2 credentials.
  • The server log. The INFO-level line for CONFIG SET printed primaryauth and the TLS client key passphrase.

  They are now shown as (redacted).

Before you upgrade

• Upgrade replicas before their master. A 1.2.0 master sends transactions to its replicas wrapped in MULTI/EXEC.
  • A 1.1.x replica still applies their writes one by one, but logs two ERR unknown command warnings for each transaction.
  • A 1.2.0 replica of a 1.1.x master is fine.
  • The KiviDB Kubernetes operator already restarts a shard's replicas first and its master last.
• No going back to 1.1.x once 1.2.0 has saved. A snapshot holding a time series with compaction rules uses a format 1.1.x cannot read. Take a backup before you upgrade if you may need to go back.
• With the AOF on, writes are slower. Waiting for the record costs 20–28% of SETs a second with 50 clients, and under 10% with one client; reads cost nothing. aof-ack-after-write no gives 1.1.x's behaviour and speed back, at the cost of losing the last few acknowledged writes to a killed process.
• Scripts: pcall and assert behave as in Redis.
  • A script that read e.err from an error pcall caught finds the message in e itself now.
  • assert with a table as its message fails with bad argument #2 to 'assert'.
• CONFIG SET refuses what it cannot set, as Redis does. Unknown names, and settings that only take effect at startup, are now errors. Before, they replied OK. Settings that used to be ignored in the configuration file are now applied.
• SORT with a BY or GET pattern needs read access to every key, as in Redis. A user whose ACL names some keys gets ERR GET option of SORT denied due to insufficient ACL permissions. for GET * or GET *->field, and the same for a BY with a *. GET # and BY nosort still work.
• AUTH <password> is refused when no password is set, as in Redis. Before, it said OK.
• Search scores changed, to Redis's. A word now also finds its other forms (stemming), and WITHSCORES / ADDSCORES values are Redis 8.10's: a TAG clause adds to the score, a field's WEIGHT counts each occurrence instead of multiplying the score. A threshold set on the old scores needs adjusting. VERBATIM on a query, or NOSTEM on a field, keeps the old matching.
• VSIM / VLINKS scores are Redis's, from 0 to 1 as (1 + cosine) / 2. A threshold an application set on the old scores, which went from -1 to 1, needs adjusting.
• activedefrag is now on by default.
• A clean shutdown pauses clients while its replicas catch up, for at most shutdown-timeout seconds (10 by default; tens of milliseconds in our tests). SHUTDOWN NOW and SHUTDOWN NOSAVE don't wait.
• repl-backlog-size is 64 MB by default, not 1 MB as in Redis and 1.1.x, so that replicas carry on after their master restarts under heavy load: 1 MB is a fraction of a second at 100k writes a second, less than a replica takes to come back, and it was then sent the whole dataset. A master keeps the backlog only from its first replica's sync until repl-backlog-ttl (an hour) after its last replica has gone, so a server that never has a replica uses none of it. With replicas it counts in used_memory and against maxmemory: on a master with a small maxmemory (tens of megabytes), set it back to 1mb. Under heavier writes, or for replicas that are away longer, give it a few seconds of writes.
• A transaction that names a key a MIGRATE is moving gets -TRYAGAIN, as a multi-key command does in Redis during a reshard. Single commands wait for the move instead. Clients retry TRYAGAIN.
• A full sync can hold up to 1 GiB more on the master while the replica loads (repl-replica-sync-max-backlog-bytes), and that memory is not counted against maxmemory. Lower it on a master close to its memory limit.
• Replicas stand for election only when they are in step, as in Redis: one restarted while its master is down no longer takes over (unless cluster-replica-validity-factor is 0), and one in the middle of a full sync never does.
• One server per data directory. A server takes a lock on kividb.lock in its directory before it reads anything there. A second one started in the same directory waits until the first exits, and says so in its log every 5 seconds. Give each server its own directory.
• A manual failover that cannot complete pauses writes for up to 10 seconds, as in Redis. Before, the pause was 5 seconds, and writes could be lost.
• Replicas with tls-replication yes tell their master their TLS port, as Redis does, even when they also have a plaintext port.

Known issues

• AOF: a power loss during a checkpoint can leave a gap in the log (#277). A checkpoint turns the log over: it syncs the old log while new writes go to the new one. The kernel may get part of the new log to the disk before the end of the old one. A power loss inside that window, which lasts up to a few seconds on a slow disk, can keep the newest writes while losing the last ones before the checkpoint, where recovery should stop at a clean cut. A process that is killed or crashes, SHUTDOWN and a restart lose nothing: only a power loss or a kernel crash is affected. In 1.1.x, the same window stalled writes instead.
• Memory freed by dropping a large vector index stays with the process. After FT.DROPINDEX and deleting its documents (a million 1536-dimension vectors in the vector benchmark), used_memory falls at once but the resident size stays where it was, until MEMORY PURGE, which gives it back in under a second. Memory limits in containers count it until then.
• redis-cli --cluster reshard under heavy writes slows the source node's writes to seconds (#310). CLUSTER GETKEYSINSLOT, which redis-cli calls for every slot, and CLUSTER SETSLOT … NODE walk the whole keyspace (#294), so a 128-slot reshard of a node with 2M keys took 12-27 minutes under 100k writes a second. Meanwhile write p99 on that node rose to about a second with --io-model uring (100 ms with tokio). No write was lost, and every reshard finished. 1.1.x does the same. CLUSTER MIGRATION IMPORT, which the Kubernetes operator uses, isn't affected: reshard with it, or run redis-cli resharding when writes are light.
• WAIT on an io_uring server answers 0 too early under concurrency. With 16 clients each doing a write and WAIT 1 200, about a fifth of the WAITs returned 0 after 200-400 ms although the replica had the write, and the server did about 200 such pairs a second, against 6,000 with --io-model tokio. No write is lost: the replica has them. 1.1.x does the same. Until it is fixed, a server that relies on WAIT can run with --io-model tokio.
• A reshard under heavy load can lose writes (#331). While slots move between nodes, a write that was routed to the source node just before its key moved can still run there after the move. This applies both to MIGRATE, as used by redis-cli --cluster reshard, and to CLUSTER MIGRATION IMPORT, as used by the Kubernetes operator. With MIGRATE, the reshard then stops on BUSYKEY, and finishing it can lose the writes made to that copy. With CLUSTER MIGRATION IMPORT, the write is deleted along with the slot. It was seen only under heavy CPU contention. A redis-cli reshard can also stop on a TRYAGAIN for a hash whose field is about to expire (#332); running it again succeeds. Until 1.2.1, reshard while writes are light, or pause writes to the slots being moved. Both are fixed in 1.2.1.

Benchmarks

benchmarks/1.2.0 has the numbers measured on the code of 1.2.0, against Redis 8.10, Redis Cluster, Valkey 9.1, Dragonfly 2.0, Garnet 2.2 and kvrocks 2.17, with the harness that produced them, the raw results and the charts: throughput and p50/p99/p99.9 latency by data type and value size, where the CPU goes, memory as a dataset grows and during a snapshot, vector search with vector-db-benchmark, the cache benchmark from #202, and smoke, soak, failover and Kubernetes operator tests. Where KiviDB loses, they say so and why.

Thanks

Thanks to the contributors whose work is in this release:
@Parikalp-Bhardwaj (writes to expiring keys that were lost, active expiry with hash tags, maxclients within the file descriptor limit, TLS handshakes that blocked new connections, XREAD BLOCK waiters, the 24-node failover tests, a race in active defrag that he found and fixed, and testing the fix for evictions during a snapshot),
@mvanhorn (COMMAND DOCS, and TIMEOUT for FT.SEARCH and FT.AGGREGATE),
@vimoppa (pcall and assert as in Redis),
@Sourabh-ctrl (HyperLogLog in Redis's format),
@Ibrahim-techie (SORT ... STORE clears the destination's TTL),
@ShryukGrandhi (COMMAND DOCS versions, and hashes loaded from a snapshot that stayed, empty, after their fields expired),
@RagedPanda (CONFIG GET threads),
@AryanSingh2029 (the Python examples' requirements), @de-ep (rejected_connections in INFO stats), @huong8373tt-beep (EXPIREAT with a negative time) and @saikishore222 (a backup test that failed on slow restarts). Thanks also to @osokin, who maintains the FreeBSD port, for the report that got KiviDB building on FreeBSD again.

What's Changed

• Release: announce stable releases in Discussions by @murtazayusuf in #101
• Docs: moving from Redis, compatibility, the Azure ramp run by @murtazayusuf in #103
• Client examples: redis-py, ioredis, BullMQ, go-redis, Jedis by @murtazayusuf in #104
• Without Lua, script commands name the build that has it by @murtazayusuf in #105
• Build on 32-bit targets (#99) by @murtazayusuf in #102
• CHANGELOG: the script-command error without Lua (#105) by @murtazayusuf in #106
• Scripts: pcall gives the err, and assert takes a string, as in Redis by @vimoppa in #92
• CONFIG SET: cluster settings take effect or are refused (#111) by @murtazayusuf in #114
• Cluster failover: rank replicas by offset; stale replicas do not stand (#107) by @murtazayusuf in #115
• Slot migration: drop the keys of an import cut short by a restart (#108) by @murtazayusuf in #116
• Replication: keep the old stream as replid2 on promotion (#109) by @murtazayusuf in #117
• Cluster: cluster-announce-client-port, a client port apart from the peer port (#110) by @murtazayusuf in #118
• tests/config.rs: every test that starts a server takes the maxmemory turn by @murtazayusuf in #122
• Cluster bus: one heartbeat connection per peer, kept open (#112) by @murtazayusuf in #119
• Shrink a shard's tables once keys have mostly left them (#113, in part) by @murtazayusuf in #120
• String values in slabs of KiviDB's own, and active defrag (#113) by @murtazayusuf in #123
• Image: distroless without OpenSSL, clearing 12 of the 15 image-scan alerts by @murtazayusuf in #164
• Fix the data-loss and wrong-data issues found while writing the docs by @murtazayusuf in #163
• Fix configuration, operations and monitoring issues (#142, #143, #150, #156, #136, #149, #152, #158) by @murtazayusuf in #166
• Fix Redis compatibility issues found while writing the docs (#124-#161) by @murtazayusuf in #167
• COMMANDS.md: match the server (#125, #131, #133, #141) by @murtazayusuf in #168
• AOF acknowledges after the write; ts-duplicate-policy, Lua struct, measured INFO fields, binary channel names by @murtazayusuf in #169
• Clean up XREAD BLOCK waiters when a read ends by @Parikalp-Bhardwaj in #175
• Wait table: remove a stream reader's registrations in constant time (#170 follow-up) by @murtazayusuf in #177
• FT.SEARCH and FT.AGGREGATE honour TIMEOUT and ON_TIMEOUT by @mvanhorn in #88
• Search timeouts: no limit by default, settings, rolling-upgrade safe, bounded pipeline (#88 follow-up) by @murtazayusuf in #178
• examples: VectorDBBench, KiviDB 1.1.2 on Cohere 1M by @murtazayusuf in #179
• Release: signed build provenance (artifact attestations) for every file and image by @murtazayusuf in #180
• VSIM scores as Redis; DEBUG POPULATE, LMOVE event order, MEMORY DOCTOR advice (#171, #173) by @murtazayusuf in #181
• Transactions and multi-write scripts reach replicas and the AOF as one step (#172) by @murtazayusuf in #182
• TLS: tls-port live, port 0 for TLS only, tls-auth-clients-allowed-names by @murtazayusuf in #183
• Manual failover: won while the master still holds writes, or given up (#185) by @murtazayusuf in #186
• Vector benchmark: floor 0.75x Redis 8.10 (was 2.0x, set against Redis Stack 7.4) by @murtazayusuf in #188
• Cluster: CONFIG GET no longer deadlocks the node with the bus by @murtazayusuf in #190
• AOF: a write over TLS is answered at once, not at the next once-a-second sync by @murtazayusuf in #191
• A graceful shutdown no longer loses writes it acknowledged by @murtazayusuf in #189
• Replication: a master drops a replica that has gone silent, after repl-timeout by @murtazayusuf in #193
• Replication: a replica continued by a promoted master no longer freezes after its next partial resync by @murtazayusuf in #194
• io_uring: CONFIG SET port opens and closes the plaintext port by @murtazayusuf in #195
• test(cluster): add 24-node failover coverage by @Parikalp-Bhardwaj in #184
• Memory per key, step 1: keyspace shards behind ShardMap by @murtazayusuf in #196
• Memory per key, step 2: one block per key behind an 8-byte slot by @murtazayusuf in #197
• Search: filtered scans read the clock between runs of documents, not before each one by @murtazayusuf in #198
• Vector search: SIMD distance kernels, a pre-normalised query, filtered KNN without cloned keys by @murtazayusuf in #199
• Memory per key, step 3: a key's time to live lives in its entry by @murtazayusuf in #200
• COMMAND DOCS returns summary, group, complexity and since by @mvanhorn in #86
• Memory per key, step 4: a key's LRU/LFU word lives in its entry by @murtazayusuf in #205
• Fix SORT STORE to clear destination TTL by @Ibrahim-techie in #206
• Search: TAG and NUMERIC postings, and a hybrid policy for filtered vector search by @murtazayusuf in #207
• maxmemory bounds what the process uses; a command's keys are found through a table by @murtazayusuf in #208
• Search: a filtered graph walk honours TIMEOUT; timeout tests no longer race by @murtazayusuf in #210
• Eviction: every key as likely to be sampled, and the runners-up looked at again by @murtazayusuf in #212
• Memory per key, step 5: strings written in place; a key's use written by its own lookup by @murtazayusuf in #209
• Shutdown: the io_uring ports stop listening before the process ends by @murtazayusuf in #211
• Search: a missing index is reported in Redis 8's words by @murtazayusuf in #213
• Search: stemming, Redis 8's scores, and a TEXT index of numbers by @murtazayusuf in #214
• Search: a query reads only the documents of its page, and writes never wait for queries by @murtazayusuf in #215
• Fix Redis HyperLogLog binary compatibility by @Sourabh-ctrl in #96
• Search: FT.DROPINDEX DD reaches replicas and the AOF as DELs, then the drop by @murtazayusuf in #218
• Vector search: vectors and links in blocks by node id; prefetch; lighter pruning by @murtazayusuf in #219
• COMMAND DOCS reports Redis's version as since, not Valkey's by @ShryukGrandhi in #217
• Expired keys: no lost writes, DEL propagated, nothing expires while loading by @murtazayusuf in #223
• AOF: one checkpoint at a time, and a crash during one loses nothing by @murtazayusuf in #224
• TLS: a stalled handshake no longer blocks connections; maxclients enforced by @murtazayusuf in #225
• CI: redis-benchmark compares with Redis 8.10.2, not Ubuntu's Redis 7.0 by @murtazayusuf in #226
• Tests: three races fixed on 1.1.3, brought to master by @murtazayusuf in #228
• Cluster: a master back without its dataset does not wipe its replicas by @murtazayusuf in #229
• Vector benchmark: filtered and high-dimension datasets, Dragonfly by @murtazayusuf in #237
• HNSW vectors written after a checkpoint are found again after a crash by @murtazayusuf in #238
• Replicas of a master before 1.1.3 expire keys themselves (#231) by @murtazayusuf in #239
• Search and vector sets: ADDSCORES rows, FT.EXPLAIN text, VADD errors as Redis 8.10; tests hold Redis's ranking of a word and its stems by @murtazayusuf in #246
• Manual failover no longer stalls on a write waiting out the handoff; robust Lua PTTL window by @murtazayusuf in #241
• Five small correctness fixes for 1.2.0: GETDEL/MOVE LFU, noscript commands, lazyfree-lazy-expire config, dead-key access word, expired ordering by @murtazayusuf in #248
• handler: name Valkey 9.1.3 as the no-script list's source by @murtazayusuf in #252
• Vector benchmark baseline from run 41273218810 by @github-actions[bot] in #253
• Replication: a MOVE that moved nothing is not sent; MOVE ordered where it goes by @murtazayusuf in #261
• e2e: replication tests never pair two servers on one port by @murtazayusuf in #263
• Tests: wait for the replica to drop the index after FT.DROPINDEX DD by @murtazayusuf in #264
• Search: a count leaves out a key the expiry is deleting by @murtazayusuf in #250
• CONFIG GET threads gives the --threads value; CONFIG SET with an unpaired name is a syntax error by @RagedPanda in #265
• EXPIRE/PEXPIRE: a positive time to live never deletes the key at once (replica divergence) by @murtazayusuf in #267
• docs(examples): add missing requirements.txt for the Python client examples by @AryanSingh2029 in #266
• Cluster: a master that may have been failed over takes no write until it knows by @murtazayusuf in #268
• EXPIRE family: check NX/XX/GT/LT before a time that deletes the key by @murtazayusuf in #272
• AOF: a checkpoint no longer stalls writes on a slow disk; count early acks (#243) by @murtazayusuf in #276
• Cluster: a slot migration's handoff holds writes until the source knows where the slots are (#278) by @murtazayusuf in #279
• MIGRATE: delete a key only if it still holds what was sent (#274) by @murtazayusuf in #280
• Search: order index walks against writes to their documents (#275) by @murtazayusuf in #281
• Concurrent writes to one key: order every client write, and let hot single-key writes share it (#271) by @murtazayusuf in #286
• Eviction: a snapshot no longer makes writes evict keys that fit (#284) by @murtazayusuf in #285
• Release 1.2.0 by @murtazayusuf in #289
• EXPIREAT: delete keys for negative timestamps by @huong8373tt-beep in #282
• Tests: give the sealed backup restart test a 10 s TTL (#269) by @saikishore222 in #292
• Update stats::rejected_connections when a connection is rejected by @de-ep in #293
• Hashes loaded from dump.kdb are watched for field TTLs by @ShryukGrandhi in #304

New Contributors

• @Parikalp-Bhardwaj made their first contribution in #175
• @mvanhorn made their first contribution in #88
• @Ibrahim-techie made their first contribution in #206
• @ShryukGrandhi made their first contribution in #217
• @RagedPanda made their first contribution in #265
• @AryanSingh2029 made their first contribution in #266
• @huong8373tt-beep made their first contribution in #282
• @saikishore222 made their first contribution in #292
• @de-ep made their first contribution in #293

Full Changelog: v1.1.3...v1.2.0