Repository navigation
Releases
v0.9.8
Memcache gets a crash fix and now answers like memcached, and CI runs the functional tests against memcached too.
Upgrade notes
- memcache
casis refused with more than one rack. Each replica's memcached numbers its items itself, so acascould change only some replicas and leave them disagreeing. With more than one rack, Dynomite now answerscaswithSERVER_ERRORand drops acas noreply. With one rack,casworks as before. UnderDC_SAFE_QUORUMandDC_EACH_SAFE_QUORUM,getscan fail when the replicas' cas uniques differ. - Memcache errors no longer close the connection. Unknown or unsupported commands (
gat,stats,flush_all,version, meta commands, upper-case commands) getERROR. Empty lines and known commands without their arguments getCLIENT_ERROR bad command line format, as current memcached sends. In both cases the connection stays open. Before, Dynomite closed it. - memcache
quitcloses without a reply. It used to send+OK, which is Redis's reply. RedisQUITstill gets+OK. - Redis
MGETwithout a quorum now answersFailed to achieve Quorum, the same asGET, instead of-ERR unknown: unknown.
Fixes
- A memcache
getorgetsthrough any node outside the first rack crashed Dynomite, even with a single key. - A memcache
getthat failed in part (for example, no quorum) closed the client's connection with no reply, and the requests after it were lost. It now returns the error, and the connection goes on. - Memcache syntax that memcached accepts no longer closes the connection: a negative expiry,
deletewith its old time argument, andincr,decrortouchwith a value that isn't a number.
Documentation
- The README describes the memcached test coverage. It also notes that replicas can disagree after an
incrordecrthat makes the number shorter, because memcached only sometimes pads the result with spaces.
Testing
- The functional tests can run against memcached (
DYN_TEST_DATASTORE=memcache). They compare replies byte for byte with a standalone memcached's and covernoreply, key and value limits, flags,incr/decrlimits, expiry, hash tags and pipelining. CI runs them against memcached 1.6.45. - The Valkey and Redis matrix now tests only their latest releases, so CI has 8 functional test jobs instead of 16.
- Local
actruns can use their own container image (CI_IMAGE), which cuts a full local run from about 6 minutes to 160 s. GitHub runs are unchanged.
Full Changelog: v0.9.7...v0.9.8
v0.9.7
Valkey becomes the recommended backend, newer Redis and Valkey commands are supported, and two replication and read repair bugs are fixed.
Upgrade notes
- SUNIONSTORE, ZINTERSTORE, ZUNIONSTORE, and SORT and GEORADIUS(BYMEMBER) with STORE or STOREDIST are writes now: they run in every rack and DC under the write consistency level. Before, Dynomite took them for reads, so the result was stored only in the local rack (DC_ONE) or the local DC (quorum levels); copies of those destination keys elsewhere may be stale or missing from before the upgrade. TTL is a read now, like PTTL.
- ECHO goes to the local datastore, like PING, instead of getting the unknown command error.
- Commands that only newer datastores have (see below) are passed on, and a datastore without one refuses it. Upgrade all of a cluster's datastores before using such a command, or a write runs on some replicas and is refused on others.
Backends
- Valkey is now the recommended and default-tested backend; Redis stays fully supported through the same protocol (RESP,
data_store: 0), and the example configurations work with either. See the README's new Backends section for why (Redis is no longer BSD-licensed since 2024). - CI runs the functional tests against the latest Valkey and Redis releases (9.1.3 and 8.10.3) and Debian trixie's packages (Valkey 8.1.1, Redis 8.0.2).
New commands
- Hash field expiry (Redis 7.4 and 8, Valkey 9): HSETEX, HGETEX, HGETDEL, HEXPIRE, HPEXPIRE, HEXPIREAT, HPEXPIREAT, HPERSIST, HTTL, HPTTL, HEXPIRETIME, HPEXPIRETIME.
- DELIFEQ (Valkey 9) and ECHO.
- GETEX, GETDEL, EXPIRETIME, PEXPIRETIME, LPOS, SMISMEMBER, ZMSCORE, ZPOPMIN, ZPOPMAX.
- GETEX and HGETEX are always writes, since they can change TTLs. ZPOPMIN and ZPOPMAX pop the same members on every replica, so unlike SPOP they work with any number of racks. Read repairs don't cover the new commands. notes/redis.md has the details, including replies in milliseconds that can differ between replicas after a relative TTL.
Fixes
- The results of the commands listed in the upgrade notes are replicated to every rack and DC.
- With read repairs enabled, SELECT, SCRIPT EXISTS/FLUSH/KILL, CLIENT SETINFO, HELLO, DBSIZE and
dyno_configread past the end of an internal table, and whatever followed it decided whether the request was rewritten for read repair.
Testing
- The functional tests run on Valkey or Redis (
DYN_TEST_SERVERpicks the server binary) and print which one they use. - New functional tests for the stored results, the new commands, and the read repair table; the parser unit tests also check whether a request is taken for a read or a write.
- A failing functional test job in CI prints the end of every node's log and uploads the full logs as an artifact.
Full Changelog: v0.9.6...v0.9.7
v0.9.6
A testing and CI release: Dynomite itself is unchanged since v0.9.5, apart from its version. The functional tests are deterministic and much faster, and CI runs them in parallel.
Testing
- The read repair test is deterministic: it used to depend on which replica answered first, and could fail under AddressSanitizer. It now holds back the stale replicas' replies so the read sees all of them, and checks that every stale replica gets the newer value and timestamp.
- No fixed waits: the tests wait until a new cluster's nodes are connected to their datastores and to each other, and the expire test uses a 300 ms PEXPIRE and polls instead of sleeping 7 s.
- The PING test, which doesn't depend on the cluster's layout, runs once per run of
test/run_tests.sh.test/run_tests.shtakes test scripts as well as cluster files, so a part of the suite can run alone. - The functional tests take ~85 s instead of ~214 s in a local container.
Build and testing
- CI runs the functional tests, plain and under the sanitizers, in four parallel parts each: one per cluster file, and the peer TLS and seed resolution tests. A CI run takes ~1.5 min instead of ~4.5 min.
- The sanitizer build uses
-Werror, like the other build jobs; the last GCC warnings it had, in the unit tests, are fixed.
Full Changelog: v0.9.5...v0.9.6
v0.9.5
Seeds that resolve later or move, two memory fixes found by AddressSanitizer, and more build and test coverage in CI.
Upgrade notes
- A seed in
dyn_seedswhose name doesn't resolve no longer makes the configuration invalid: the node starts with a warning and that peer counts as down until its name resolves. A typo in a seed name now shows up as a down peer and repeated "address resolution ... failed" log lines, not as a refused start. - A peer's name is looked up again before every connection attempt, so a node follows a peer whose name moves to another address (logged as "peer '...' resolves to ... now"). The lookup blocks the node while it runs: with a slow or unreachable DNS server, a node can stall for up to the resolver's timeout, once per reconnect backoff. Seeds given as IP addresses aren't affected.
- A node now starts even when connecting to one of its peers fails at once; before, that stopped the node, and peers listed after the failing one weren't connected at startup.
Fixes
- Read repair: when the stale reply came from the local datastore, the repair wrote a byte past the end of a heap block. And a stale reply from a peer listening on different ports than the coordinator marked that peer as local for good, so requests meant for it went to the local datastore.
- Command names in the Redis and memcache parsers, the MD5 and Hsieh hashes and an internal hash table read 2- and 4-byte words at unaligned addresses, which is undefined behaviour and can fault on CPUs other than x86. They are loaded safely now, with the same results and speed on x86.
- The
hash_tagconfiguration string was never freed. - A gossip peer update with no nodes returned an uninitialized status.
Testing
- CI also builds with clang and for 32-bit x86 (with
-Werror, running the unit tests), and runs the unit and functional tests under AddressSanitizer and UndefinedBehaviorSanitizer, which found the two memory fixes above. - A new functional test,
test/seed_test.py, starts a node with a seed whose name doesn't resolve yet, then makes it resolve and moves it to another address.
Full Changelog: v0.9.4...v0.9.5
v0.9.4
Quorum fixes for datacenters with more than 3 racks, a stats /ping that reflects the datastore, working TCP keepalive timing, and Redis commands with their newer optional arguments.
Upgrade notes
/pingon the stats port now answers 503 Service Unavailable ("Datastore is not connected") while the node has no connection to its datastore, instead of always OK. Health checks using it will now fail in that state, and pass again by themselves once the node reconnects.- A datacenter can have at most 8 racks: a node whose seeds list more doesn't start, and a peer found later in a 9th rack is left out, with an error in the log.
CLIENT SETINFOaccepts onlylib-nameandlib-ver; other attributes get Redis's error.- Peer and datastore connections now really use a 15 second keepalive: an unresponsive peer is dropped after about 75 seconds instead of the system's default (2 hours idle on Linux).
- The log file is created with mode 0644 and the daemon runs with umask 022 (they could end up world-writable before).
- A client dropping a stats request before the reply is logged as a rate-limited warning (once per 10 seconds, with a count) instead of an error each time.
Fixes
- With more than 3 racks in a datacenter,
DC_QUORUMand the stricter consistency levels could hang requests, return a minority's reply, or crash the node. A quorum is now more than half of the racks giving the same reply, counted over all replies, including per element forMGETand other multi-key replies. - The 15 second TCP keepalive was asked for but only
SO_KEEPALIVEwas set, so NATs and firewalls that drop idle connections could silently cut quiet peer links.TCP_KEEPIDLE,TCP_KEEPINTVLandTCP_KEEPCNTare set now. - Redis commands with newer optional arguments were refused with "wrong number of arguments" or closed the connection:
EXPIRE/EXPIREAT/PEXPIRE/PEXPIREATwithNX|XX|GT|LT,ZRANK/ZREVRANKwithWITHSCORE,LPUSHX/RPUSHXwith several elements,RESTOREwith its options, multi-hashSCRIPT EXISTS,SCRIPT FLUSH ASYNC|SYNC,QUITwith an argument and multi-sectionINFO. Checked against Valkey 9.1.3 and 8.1.1 and Redis 8.0.2. - The "FORWARD FAILED" log line showed the wrong error text, and requests whose rewrite or split failed got "Unknown error -1"; they get "Unknown Error" now.
- On 32-bit builds, log lines with request ids, timestamps and counts were printed wrong. A peer message type past the last known one is now a parse error instead of being kept or wrapped.
Build and testing
- Warning-free builds on 32-bit and with clang.
- New unit and functional tests for quorum with up to 8 racks, keepalive timers,
/pingand dropped stats requests, the Redis optional arguments, and the log file mode; one datacenter of theDC_EACH_SAFE_QUORUMtest cluster has 4 racks now.
Full Changelog: v0.9.3...v0.9.4
v0.9.3
This release is about commands that silently did the wrong thing, or broke the connection, across nodes and replicas: they now work or get a clear error.
Upgrade notes
- Multi-key commands spanning nodes are refused. RPOPLPUSH, SMOVE, SDIFF, SINTER, SUNION and their STORE forms, ZINTERSTORE, ZUNIONSTORE, PFCOUNT, JSON.MGET, EVALSHA, and SORT and GEORADIUS with STORE go whole to the node of their first key. When their keys belong to different nodes in any rack, they now get
-ERR Dynomite: Keys in the request cannot span multiple nodes, use a hash tag. Before, they read keys the node doesn't have and wrote destinations no later read would find. Keys sharing a hash tag work as before, and so does any setup with one node per rack. EVAL was already checked, but only in the local rack; now every rack is checked. - SPOP is refused when there's more than one rack, because each replica popped different random members and the replicas' sets diverged. Use SRANDMEMBER and then SREM instead. With a single rack, SPOP works as before.
- SRANDMEMBER, SDIFF, SINTER and SUNION read from the local rack only, at every consistency level. Their replies differ between replicas (random members, or members listed in each replica's own order), so quorum reads of them always failed.
- Unsupported commands get an error and the connection stays open. XADD, FLUSHALL, BLPOP, CLIENT LIST and other commands Dynomite doesn't support now get Redis's
unknown commandorunknown subcommanderror. Before, the connection was closed without a reply. - Big-endian machines hash keys like x86. murmur now gives the same tokens on every architecture. Tokens on x86 and arm64 don't change. A cluster running only on big-endian nodes (s390x, for example) would see its keys move.
- Log changes: the "no quorum" error is reworded, and it and the DC_ONE "Swallowing cross rack error" line (now a warning) are logged at most once every 10 s, with a count of the lines left out. Unsupported commands from clients log a rate-limited warning. Update anything that matches on these lines; the stats counters still count every request.
Fixes
SELECT 0, which some clients send on connect, is answered with+OKinstead of closing the connection. Other database indexes get the error Redis gives with one database.LPOP,RPOPandSRANDMEMBERaccept a count.PFCOUNTaccepts several keys.UNLINKworks; before, it closed the connection. It's sent asDEL, which gives the same reply.DBSIZEis supported. It counts the keys of the node's own datastore, like KEYS and SCAN.- murmur and jenkins no longer read the key as native, possibly unaligned words. jenkins no longer reads up to 3 bytes past the key, and it's faster for aligned keys.
Documentation
notes/redis.mdgains a section on commands with more than one key and on hash tags, and documents SELECT, DBSIZE, UNLINK, SPOP and SRANDMEMBER, and how unsupported commands are answered.
Testing
make checkpins every hash function's tokens and checks they don't depend on the key's alignment. It now passes on x86, arm64 and s390x.- New unit test for the log rate limiting, and parser tests for the new and changed commands.
- New functional tests for SELECT, the pop counts, SPOP and SRANDMEMBER across racks, multi-key routing with and without hash tags, DBSIZE and unsupported commands. The test clusters now set
hash_tag: "{}".
Full Changelog: v0.9.2...v0.9.3
v0.9.2
A documentation and polish release: no behaviour changes beyond corrected log and help texts.
Upgrade notes
- Some log lines, the stats descriptions (
-D), the hash tool's usage text and configure's error messages read differently after the spelling fixes. Error replies to clients, stats keys and configuration option names are unchanged; if you match log text, check your patterns.
Fixes
- dyno-hash-tool reported the wrong file when the output file failed to open.
- A seed with a bad port was reported against the datastore format instead of the seed format
hostname:port:rack:dc:tokens [name]. - Missing spaces in the read_consistency/write_consistency and server error messages.
Documentation
- README,
data_store: what memcached support covers (ASCII text protocol, the supported commands), that replies from replicas aren't reconciled and there are no read repairs, and that the functional tests run against Redis only. - README,
hash: keep the defaultmurmur(Dyno's token-aware routing depends on it) and choose it once, since changing it remaps nearly every key. Avoidcrc32: libmemcached's variant only gives values from 0 to 32767, so all keys land on one node (crc32ais the standard CRC-32). - README,
max_msgs: how a node behaves under load: pausing busy clients and peers, closing clients over max_msgs, and refusing requests to a peer with 20000 queued. - About 200 spelling and grammar fixes across comments, messages, docs and build files.
Full Changelog: v0.9.1...v0.9.2
v0.9.1
This release is about behaviour under heavy load and when nodes are down: no more dropped connections from long queues, and no more requests left without a reply.
Upgrade notes
- A node no longer closes a client, datastore or peer connection because too many requests are queued on it. Instead, a client or peer sending requests is paused at 20000 outstanding requests (TCP pushes back) and resumed at 10000.
max_msgsnow limits only new client requests: a client that would go over it is closed, with a warning in the log. Replies and requests from peers are always accepted.- A peer connection with 20000 requests queued refuses new ones with the new error
-ERR Peer: Peer Node has too many requests queued, counted in the new statspeer_queue_full_requestsandremote_peer_queue_full_requests. - Log changes: the startup lyrics and ASCII logo are gone (they put seven lines without a timestamp into journald and docker logs), and the exit line
done, rabbit doneis nowdynomite stopped. Update anything that matches on them.
Fixes
DC_EACH_SAFE_QUORUM: with a remote DC down, or too few of its racks left for a quorum, requests never got a reply, and pipelined replies after them were stuck too. They now getPeer Node is not connected(or the error that stopped the forward) at once.- One client pipelining more than 20000 requests was reset after about 100 replies; many clients pipelining through one node got
Datastore refused connection. Both now get every reply. - A stalled peer no longer makes its whole queue fail: requests already queued are replicated once it runs again.
- Replicas that disagree under
DC_SAFE_QUORUMandDC_EACH_SAFE_QUORUMcould crash the node when no message was free; the "Failed to achieve Quorum" error is now always built. - At the
max_msgslimit, replies from the datastore and from peers were refused too, which stalled or closed connections with unrelated errors. - Every close of a datastore or peer connection, and every failed reconnect, logged a spurious
epoll ctl ... failed: No such file or directoryerror.
Testing
- New unit tests (
make check) for the quorum error and themax_msgslimit at an empty free list. - New functional test that stops one rack of a DC and then all of it under
DC_EACH_SAFE_QUORUM.
Full Changelog: v0.9.0...v0.9.1
v0.9.0
This release adds mutual TLS between nodes and starts the vX.Y.Z version scheme.
New: mutual TLS between peers
- With
peer_tls: true, every connection between nodes (thedyn_listenport, gossip included) is mutual TLS 1.3: both sides present a certificate and verify the other's against the cluster CA, and a connection without a valid certificate is closed before any message is read. - Configure it with
peer_tls_cert,peer_tls_keyandpeer_tls_ca. Bad files refuse to start, anddynomite -treports them. Dynomite warns at startup if its own certificate won't verify, and daily when it expires within 14 days. - All nodes must switch together. The client port, the stats port and the datastore connection stay plaintext.
- See "Securing traffic between nodes" in README.md.
Upgrade notes
- Building needs OpenSSL 3 (
libssl-devon Debian). - Versions are now
vX.Y.Z; this is the release after v0.6.22-12.
Fixes
- A
dyn_seedsentry rejected by the configuration parser no longer leaks memory.
Testing
- The functional tests run every cluster both plaintext and with mutual TLS, plus dedicated TLS tests.
Full Changelog: v0.6.22-12...v0.9.0
v0.6.22-12
A fix for running Dynomite in containers and under service managers.
Fixes
- Dynomite exits cleanly, with status 0, on SIGTERM and SIGINT and removes its pid file. Before, SIGTERM killed it on the spot, and as PID 1 in a container it ignored SIGTERM, so
docker stopandkubectl delete podwaited out their grace period and then killed it.
Full Changelog: v0.6.22-11...v0.6.22-12