Skip to content

v1.2.1

Choose a tag to compare

@github-actions github-actions released this 12 Nov 09:12
· 1080 commits to master since this release
255ad15

KiviDB v1.2.1

Upgrade to 1.2.2. It fixes GHSA-8rm8-p5c5-g3mr, which affects 1.2.0 and 1.2.1: with a search index, a multi-key write over an expired key could stop the server answering.

What's in 1.2.1

1.2.1 is a patch release of 1.2.0. It fixes the 1.2.0 known issues of reshards that could lose writes and replicas that could stop while they saved, and many smaller ways in which a replica, the append-only file or a reshard could end up with other data than the master:

  • Clusters.
    • A reshard under load no longer loses acknowledged writes, with MIGRATE (redis-cli --cluster reshard) or with CLUSTER MIGRATION IMPORT (the Kubernetes operator): a write checks inside the write gate whether its keys have left the node, and is sent on with -ASK or -MOVED if they have. The same holds for EXEC and scripts, and for every other way a slot leaves a node (#331, #337, #339, #407).
    • A failover of the source during an atomic slot migration no longer loses the writes the new master acknowledged: a failover and a handover now exclude each other (#516).
    • A drained node no longer deletes the keys it moved from its new copy of them (#348), keys written to a slot that came back are no longer deleted with its old keys (#353), and writes no longer stay paused after a handoff (#349).
    • MIGRATE no longer stops redis-cli --cluster reshard with TRYAGAIN for an unchanged hash whose field is about to expire (#332).
    • Cluster admin commands (CLUSTER FLUSHSLOTS, ADDSLOTS, ADDSLOTSRANGE, BUMPEPOCH, SET-CONFIG-EPOCH, COUNT-FAILURE-REPORTS, GETKEYSINSLOT) answer as in Redis (#552, #553, #563), and a restarted node gives its own address at once (#307).
  • Replicas and the AOF stay in step with the master.
    • A replica no longer stops for good while it saves or is promoted (#373).
    • CLIENT PAUSE ... WRITE replies only once every write already being applied is in the offset, so a switchover (the operator's, and any that works the same way) no longer promotes a replica without the last acknowledged write (#629).
    • A write to a key past its TTL first deletes it and sends its UNLINK on, so a replica no longer keeps a key the master no longer has (#628). A write that changed nothing is no longer sent on, as in Redis.
    • A hash field past its TTL is removed by the master only, and sent on as an HDEL (#415). Its deadline is kept to the millisecond everywhere, and a replica holds the master's (#594, #489). A relative TTL reaches replicas and the AOF as the master's own deadline (#602).
    • A write that fails changes nothing (JSON.MSET, XGROUP CREATE ... MKSTREAM, TS.ADD) (#551), SPOP with a count from a RESP3 client is replicated, and a write in database 0 no longer reaches replicas and the AOF in another database (#423).
    • A replica that falls behind no longer makes its master evict its keys and refuse writes (#542).
    • A master that becomes a replica ends its blocked clients, as Redis does (#591). On --io-model tokio, CLIENT KILL ends a replication link (#595). WAIT and WAITAOF no longer starve socket I/O on io_uring (#498).
    • Sentinel no longer promotes a replica set to replica-priority 0, and new --replica-priority and --replica-announced flags set them (#443).
  • Durability.
    • appendfsync always syncs each write to disk before it is acknowledged, as in Redis (#513). It was accepted and did nothing.
    • A restart with the AOF on after a save with it off no longer applies writes twice (#354), and CONFIG SET appendonly yes starts a new log instead of appending to an old one (#350, #368, #359). A log that is not replayed is never deleted: it is kept aside in dir as superseded.<time>.<n>.appendonly.aof, and counted in INFO persistence (aof_superseded_files, aof_superseded_bytes).
    • With appendonly no, no appendonly.aof is made or needed (#47).
    • A snapshot or RDB file that is not whole (cut short, a wrong checksum, a Dragonfly .dfs file, databases the server does not have) is refused instead of loaded in part (#444, #474, #477, #482).
  • Faster where 1.2.0 was slow.
    • INFO no longer stalls writes on a server with many keys that have a TTL: about 0.1 ms with 5 million of them, against 55 ms (#622).
    • An idle server with many keys that have a long TTL no longer spends CPU on them: 0.5-0.75% of a core with 5 million keys, against about 40% (#618).
    • HSET no longer takes a lock every worker shares when no search index is defined (#544). ZREM, ZCOUNT and ZREMRANGEBYSCORE no longer scan the whole sorted set (#386, #408).
  • Moving data in. KiviDB reads Valkey 9.0's and 9.1's RDB format: a dump.rdb, --preload-file, a full sync from a Valkey 9 primary and RESTORE of its DUMP payloads (#449). Dragonfly's hashes with field TTLs and JSON documents, KeyDB's field TTLs and DUMP payloads, and Redis Stack 6.2's files load; what KiviDB does not read is refused by name (#452). An RDB file's databases past 15 load into their own databases (#477).
  • Keys are bytes, as in Redis. Every command takes a key that is not UTF-8, as do stream group and consumer names, vector-set elements, KEYS and SCAN patterns and passwords (#437, #446).
  • Compatibility.
    • HELLO reports the Redis release KiviDB follows (server: redis, version: 8.10.0), so redis-py and Jedis turn on their client-side caches (#451).
    • A client that goes while it is blocked no longer takes the next element, which was lost with Bull and n8n's queue mode (#384). BRPOPLPUSH and BLMOVE time out with *-1 (#385).
    • A Lua script's false is a RESP3 null to a RESP3 client, so Celery chords complete with redis-py 8 (#431).
    • @connection holds what it holds in Redis, so redis-py's cluster client connects with +@connection (#392).
    • MULTI, WATCH, HELLO, the ACL log, TTL rounding, expire-time bounds, blocking commands in scripts, SWAPDB, DEBUG, cluster, JSON, search and time series commands answer as Redis, RedisJSON, RediSearch and RedisTimeSeries do. JSON numbers keep every digit (#445).
  • Installing.
    • Signed apt and dnf repositories at https://packages.kividb.io, for amd64 and arm64: apt install kividb and dnf install kividb. See docs/INSTALL.md.
    • The .deb and .rpm say they need glibc 2.34.
    • The release is published on crates.io: cargo install --locked kividb, or cargo binstall kividb.
    • Every release has a source tag, X.Y.Z-src, and source archives are small again.

CHANGELOG.md has the full list.

Security

1.2.1 fixes two issues; upgrade from any 1.1.x or 1.2.0.

  • Commands that write only with some arguments are checked as writes (GHSA-cg4w-p5w5-57w5, high).
    • GEORADIUS/GEORADIUSBYMEMBER ... STORE|STOREDIST and MIGRATE were let through by read-only key permissions such as %R~*.
    • +@read granted them, and -@write didn't remove them or the FUNCTION LOAD|DELETE|FLUSH|RESTORE subcommands.
    • min-replicas-to-write, the out-of-memory refusal and aof-ack-after-write didn't apply to them.
    • They are now refused with NOPERM, NOREPLICAS or OOM, as in Redis.
    • Affects 1.0.0 to 1.2.0.
  • Deeply nested JSON documents could crash the server (GHSA-qh83-449g-hwm5, medium).
    • Repeated JSON.SET at ever deeper paths could build a document thousands of levels deep. Reading or copying it then overflowed the stack and stopped the process.
    • A write that would nest a document 128 levels or deeper is now refused with ERR recursion limit exceeded, as RedisJSON does.
    • Snapshots and RESTORE payloads holding such a document, or an array index past the largest allowed, are refused.
    • Affects 1.1.0 to 1.2.0.

Before you upgrade

  • Upgrade replicas before their master, as always. 1.2.1 can write what 1.2.0 cannot read:
    • A 1.2.0 server refuses to start on a snapshot that holds a key, a consumer group or consumer name, or a vector-set element name that is not UTF-8, or the empty key, and says so. A 1.2.0 replica cannot full-sync from a 1.2.1 master that holds one, and drops a replicated write of such a key. A 1.2.0 node refuses such a key in RESTORE, so MIGRATE to it fails and the source keeps the key.
    • A 1.2.1 replica of a 1.2.0 master is fine.
  • No going back to 1.2.0 once 1.2.1 has saved such a key. Take a backup before you upgrade if you may need to go back.
  • Hash fields expire on replicas when the master says so. A replica, and the AOF as it is replayed, keep a field past its TTL until the master's HDEL; a replica hides it from its readers meanwhile.
    • While any node of a master and its replicas is 1.2.0, fields are still removed by each node's own clock. The gap closes once the master, upgraded after its replicas, is 1.2.1.
    • An AOF written by 1.2.0 holds no HDEL for the fields that expired. Run BGREWRITEAOF once after upgrading.
    • A SORT whose BY or GET pattern reads a hash field now runs alone, as a script does.
  • WAIT with no replicas waits out its timeout, as Redis does: WAIT n timeout with n above 0 answered at once, and now answers 0 after timeout. WAIT is refused on a replica, and a negative timeout is refused. WAIT 0 answers the number of replicas that have what was written, not the number connected.
  • HELLO says server: redis and version: 8.10.0. A tool that told KiviDB apart by HELLO's server should look for kividb_version instead, as in INFO.
  • GEORADIUS and GEORADIUSBYMEMBER are refused on a read-only replica, with or without STORE, as in Redis. An application that reads from replicas with them must switch to GEORADIUS_RO and GEORADIUSBYMEMBER_RO, or GEOSEARCH.
  • ACL categories are Redis's, and apply to existing users as soon as the server is upgraded:
    • A user with +@write but not +@connection (or +@all) can no longer SELECT. Add +select.
    • A user with +@read, +@keyspace or +@generic but not +@connection can no longer run WAIT or WAITAOF. Add +wait +waitaof.
    • A user of +@all -@write, or of the readonly shorthand, may now SELECT.
  • ACL LOG answers as Redis does: fields in Redis's order, entry-id (not id) counting from 0, and the ten newest entries without a count. A client that reads entries by position or by id needs to change. acllog-max-len (default 128) is now applied.
  • CONFIG REWRITE needs a config file. A server started without one now replies ERR The server is running without a config file. To use it in the Docker image, start the server with --configfile pointing at a file that exists. On a host where an earlier version created /etc/kividb/kividb.conf this way, look for a replicaof line nobody wrote, and delete or fix it.
  • Delete sorted sets with a NaN score before upgrading. No command makes one, but a 1.2.0 server holding one (from a crafted RESTORE or snapshot) won't start on 1.2.1 if the key is in its snapshot or AOF preamble. ZRANGE key 0 -1 WITHSCORES shows nan.
  • A search index is seen only from its own database. FT._LIST lists the selected database's indexes, and the commands that name an index of another database reply Index not found. An FT.ALTER, FT.ALIASADD, FT.ALIASDEL, FT.ALIASUPDATE or FT.SYNUPDATE that a 1.2.0 AOF or master holds for another database's index is refused on replay. Once the master is upgraded, check such an index with FT.INFO from its own database and repeat what is missing.
  • Time series.
    • DUMP, RESTORE and MIGRATE leave out a series' compaction rules and source, as in RedisTimeSeries. An atomic slot migration from a 1.2.0 node, as in a rolling upgrade, loses the moved series' rules: make them again with TS.CREATERULE afterwards.
    • The options take RedisTimeSeries's spellings (CHUNK_SIZE), and a word that is not an option no longer drops the options after it. A 1.2.0 AOF replays whole, under these rules, so after the first restart a series can differ from what 1.2.0 held in memory.
    • A sample's time above i64::MAX is refused.
  • JSON.NUMINCRBY and JSON.NUMMULTBY keep a float result a float (1.5 + 1.5 is 3.0), and refuse an integer overflow with ERR numeric overflow. A client that reads such a result as an integer may need to change.
  • Expire times are checked as Redis checks them. A deadline past i64::MAX milliseconds is ERR invalid expire time, a number too large is ERR value is not an integer or out of range, EXAT 0 and PXAT 0 are refused, and SET with KEEPTTL and a time, or with two of EX, PX, EXAT and PXAT, is a syntax error. A hash field's deadline must be from 0 to 2^46 - 1 ms after the epoch. TTL and EXPIRETIME round to the nearest second.
  • In MULTI, UNWATCH, AUTH, HELLO, MONITOR, WAIT and WAITAOF are queued, as in Redis 8. MONITOR is refused there, and WAIT and WAITAOF answer at once.
  • appendfsync always now costs writes a second, as in Redis. everysec stays the default.
  • CLIENT PAUSE can answer an error (writes in progress did not finish before the pause could take hold) when a write being applied takes longer than 5 seconds, and then pauses nothing.
  • During an atomic slot migration's handoff, and for 3 s after it, CLUSTER FAILOVER on a replica of the source replies OK but does not happen. The source and the replica log why. Ask again a few seconds later.
  • Replication links in CLIENT LIST show flags=S on the master and flags=M on the replica, and connected_clients no longer counts them.
  • The packages need glibc 2.34 or newer: Debian 12, Ubuntu 22.04, RHEL 9, Amazon Linux 2023 or later. On Debian 11, Ubuntu 20.04, RHEL 8 or Amazon Linux 2 the package manager refuses them.

Known issues

  • AOF: a power loss during a checkpoint can leave a gap in the log (#277), as in 1.2.0. Only a power loss or a kernel crash is affected.
  • Memory freed by dropping a large vector index stays with the process until MEMORY PURGE, as in 1.2.0.
  • redis-cli --cluster reshard under heavy writes slows the source node's writes to seconds (#310), as in 1.2.0. No write is lost. CLUSTER MIGRATION IMPORT, which the Kubernetes operator uses, isn't affected.

Benchmarks

benchmarks/1.2.1 measures 1.2.1 against the 1.2.0 release binary, with the 1.2.0 suites' method and harnesses: throughput and latency by data type, memory per key, instructions per command, vector search, and memory during a snapshot. benchmarks/1.2.0 has the comparison with other servers.

Thanks

Thanks to the contributors whose work is in this release:
@Parikalp-Bhardwaj (hash fields expired by the master, MEMORY USAGE of sets, hashes and sorted sets, ZREM, ZCOUNT and ZREMRANGEBYSCORE without a scan, active defrag's relocation made safe against a panic, and a test of PFMERGE and PFADD),
@Ayege (JSON.NUMINCRBY and JSON.NUMMULTBY floats and overflow),
@matheussilvano (SCRIPT HELP, latency-tracking-info-percentiles, INFO commandstats, and the LangChain, LangGraph, LlamaIndex and Mem0 harnesses),
@paddor (WAIT and WAITAOF on io_uring),
@Sourabh-ctrl (SCORE and SCORE_FIELD as in Redis),
@saikishore222 (DEBUG),
@bfs2021 (the eviction tests), @ZYZ666-RGB (the LlamaIndex harness) and @Anandb71 (the chat compatibility checks).

Binary variants

Each OS/architecture ships four binaries — pick the one that matches your needs:

Tarball suffix Features included When to use
(none) — Minimal / constrained environments
-lua Lua scripting EVAL / EVALSHA
-tls TLS termination Encrypted client connections
-full Lua + TLS Recommended general-purpose build

Full changelog: CHANGELOG.md

License: GNU Affero General Public License v3 (AGPL-3.0-only)

Code of conduct: CODE_OF_CONDUCT.md


Changes since v1.2.0:

What's Changed

New Contributors

Full Changelog: v1.2.0...v1.2.1