Skip to content

Version 2.6.0

Choose a tag to compare

@fcostaoliveira fcostaoliveira released this 11 Nov 13:26
· 2 commits to 2.6 since this release

What's Changed

memtier_benchmark 2.6.0 adds --request-timeout, a sliding-window tail-latency row, a Memcached meta protocol mode and a native Windows build, and fixes many --test-time, reconnect, retry and cluster edge cases.

Read Upgrade notes first: several changes move benchmark numbers or reject command lines that 2.5.x accepted.

Upgrade notes

Command lines and exit codes

  • --request is ambiguous (#903). It now abbreviates both --requests and --request-timeout; use the full name.
  • Empty and out-of-range numbers are errors (#699, #721, #748, #761, #668). Empty values such as an unset --max-reconnect-attempts=$VAR or --max-retries=$VAR (previously 0) are rejected; pass 0 explicitly. Wrapped and overflowing values, ports outside 1-65535 and --reconnect-backoff-factor infinity or underflow are rejected too. --multi-key-get is capped at 16777214 and --data-offset at 536870911.
  • --uri (#764, #720). Credentials are percent-decoded, so a literal %XX in a password must be written %25XX; bracketed IPv6 hosts work.
  • --data-import (#767). Keys or values over 512 MiB are rejected and --generate-keys validates the file at startup.
  • Runs that exited 0 can now fail. Write errors on any output file or stdout (exit 1; a closed pipe reader such as | head is exempt) (#729, #760, #717), verification mismatches or a verification pass with no replies (#771), connection rejections counted as ops (#897; exit 1, or 2 under unlimited reconnects after --connection-stage-timeout), and a closed stdout with no --out-file (#736).
  • Signals. SIGTERM is handled like Ctrl+C (#706).

Output consumers

  • New fields. Top-level FATAL_ERROR on failed runs (#25), Totals Errors, Retry Attempts and Retried Ops with --retry-on-error (#801), Unanswered requests in Runtime (#669), and library versions (#823).
  • Changed values. Empty latency histograms keep their JSON keys and write 0.000 (#793). Percentile keys are no longer truncated or colliding (#517), duplicate --command keys get a " #N" suffix (#588, #631), and invalid UTF-8 becomes U+FFFD (#741, #751). --uri and -a values are redacted in --json-out-file and --show-config (#735).

Build and runtime requirements

  • libevent 2.0.11 or newer (#812). The default-on Prometheus exporter needs 2.1.4 or newer; on 2.1.1 to 2.1.3 use --disable-prometheus (#823). libtool is no longer required (#762).
  • A failed evthread_use_pthreads() is now a fatal error instead of a warning that could hang later (#823). memtier also warns when the loaded libevent or OpenSSL has a different major.minor than the one it was built against.

New Features

  • Request stall timeout: --request-timeout=SECS (default 0, off) fails connections whose requests are never answered, for example a proxy that completes the TCP handshake but stops calling accept(). It is a stall timeout, not a per-request deadline: a connection that keeps receiving reply data never trips it, a connection that never answers is detected after SECS, and one that goes silent after answering within SECS to 2*SECS. A timeout is a connection error (reconnected under --reconnect-on-error, otherwise the run fails). Do not set it below the length of a legitimate blocking command such as BLPOP or WAIT. (#903)
  • Sliding-window tail latency: --latency-window=SECS (default 60, 0 disables, maximum 3600) under --realtime-latencies adds a win<N>s row with the percentiles of every sample from the last N seconds. Rows show their sample count (n=), and percentiles from too few samples are flagged *. Also fixes the overall row dropping samples recorded between a client's last snapshot and its per-second reset, a bug since 2.4.0. (#386, #824)
  • Configurable progress display: --progress-interval=SECS (positive whole seconds, default 1; 0 is rejected) renders progress every SECS seconds while supervision and export keep their cadence. (#129)
  • Memcached meta protocol: --protocol=memcache_meta for SET, GET and multi-GET against Memcached 1.6.10 and newer; --wait-ratio and --command are rejected at startup in this mode. (#363)
  • Abstract Unix sockets: --unix-socket @NAME connects to Linux abstract-namespace sockets (name up to 107 bytes; use ./@NAME for a filesystem path that starts with @). (#257)
  • Native Windows build: build and run on Windows 10+ with MSYS2 UCRT64 and the autotools build. Unix sockets are rejected on Windows. (#553)
  • Cursor walks for more types: --scan-incremental-iteration follows cursors for HSCAN, SSCAN and ZSCAN as well as SCAN (standalone, single command, pipeline 1). Generated MATCH arguments now stay fixed for the whole walk, including for SCAN. (#539)
  • Structured failure summary: with --json-out-file, a failed run records a top-level FATAL_ERROR string and keeps the configuration and results already written. Successful runs and graceful interrupts do not gain the field; JSON notices on stderr appear only with --debug. (#25)
  • Retry counters in JSON: Totals gains Errors, Retry Attempts and Retried Ops under --retry-on-error. (#801)
  • RESP2 without HELLO: with --protocol=resp2, a server that answers -ERR unknown command 'HELLO' (Redis older than 6.0, other RESP servers) is treated as RESP2 instead of stalling. --protocol=resp3 still fails against such a server, now with a clearer message. (#884)

Bug Fixes

--test-time, reconnects and retries

  • Timed-run end times: clients end at the later of their deadline and last reply, send no workload past the deadline, and exit --clients-start runs at --test-time; the JSON Start time, Finish time and Total duration average all threads, not the last one. (#869, #876, #892, #722, #742, #723, #789)
  • Setup-only clients: a client that reconnects repeatedly but never submits workload no longer adds its setup interval to Total duration, and a late first connect no longer ends a --reconnect-on-error run with zero requests. (#821, #865, #714)
  • Dead server: with --reconnect-on-error and unlimited attempts (the default), the --test-time deadline wins over further reconnects and the run exits normally; a finite --max-reconnect-attempts still exits 1 when exhausted. (#666, #873, #812)
  • --connection-stage-timeout: measured in microseconds, so N can no longer abort after just over N-1 seconds; a worker that already ended is not also reported as an abort; exit 2 if any --run-count run hits it. (#860, #891, #752)
  • Hangs on rejected setup: a run whose AUTH, SELECT or HELLO is rejected on only some connections, or on reconnect, now ends or reconnects instead of hanging; a retried CLUSTER SLOTS replaces a hang. (#750, #630, #667)
  • Connection rejections and errors: -ERR Protocol error, -ERR max number of clients and -DENIED ... protected mode are not counted as operations, ordinary error replies count as connection progress (#897, #788), and connection errors and Ctrl+C of clients that never connected are reported (#707, #698, #730).
  • Retry backoff: each retry waits its own retry-backoff-ms * factor^resends; fractional factors are floored at 100 ms (or --retry-backoff-ms if smaller), nan, inf, overflowing and empty factors are rejected. (#700, #727, #694, #692)
  • Retry accounting: a --requests run no longer stops early, exits 0 having sent fewer commands than requested, or fails with "client(s) stopped before reaching their stop condition" when retries remain after the last request; in-flight requests dropped on disconnect no longer stall clients. (#559, #556)
  • Worker threads and transactions: worker threads are no longer silently re-run after connection errors (#546); a --transaction block is no longer continued after its MULTI is lost to a reconnect, transaction requests are not retried, and planned --reconnect-interval reconnects finish the rotation first (#607, #634, #615, #807, #606, #893).

Cluster mode

  • Redirects and slot maps: -MOVED retries go to the node the reply names, -ASK retries send ASKING first, empty hosts resolve to the replying connection, and CLUSTER SLOTS follows MOVED even with a full pipeline; refreshes no longer happen inside an open MULTI, twice concurrently or on every -READONLY, and back off per client on CLUSTERDOWN (1 s doubling to 8 s). (#622, #600, #787, #591, #576, #618, #662, #777, #877)
  • Connections and partial maps: only a dropped cluster connection is reconnected, and once; synchronous dial failures are retried; a refresh that retires its own connection no longer aborts the run; partial slot maps no longer leave workloads spinning, and a seed owning none of the keys no longer stalls. (#560, #574, #575, #584, #799, #612)
  • Replica reads and statistics: keyless --command requests stay off replicas unless replica reads are allowed, strict --read-preference=secondary with no replicas names the cause, retried redirects count once and arbitrary-command redirects count in MOVED/sec and ASK/sec. (#887, #901, #605, #593, #582, #594, #583, #878)

Output, JSON and statistics

  • --run-count: BEST and WORST no longer always show run 1 and prefer complete runs (a run with a failed worker carries Interrupted: true); the aggregate Totals block has a real histogram, KB/sec and miss-rate warning, and latency is weighted by completed samples. (#571, #794, #572, #581, #573, #586, #804, #757)
  • Latency and counters: built-in WAIT reports real latency in JSON (#805); run statistics and hit, miss, MOVED and ASK counters are 64-bit, so totals no longer wrap on 32-bit and Windows builds (#552, #659).
  • JSON validity: strings, keys and non-UTF-8 bytes are escaped or replaced, deep-percentile keys no longer truncate or collide, and the writer is closed on early exit, leaving parseable (possibly configuration-only) JSON, so still check the exit status. (#555, #741, #751, #517, #588, #631, #772)
  • Redaction: --uri credentials and -a are redacted (user:***, ***) in --json-out-file and --show-config, so rerunning from the JSON needs -a/--uri again. (#735, #784)
  • Prometheus: memtier_latency_seconds records every completed operation once (#665); invalid UTF-8 in labels is replaced (#763); a bind failure reports the real reason (#872).
  • --failed-keys-file and HDR output: --command requests record their key, backslashes are escaped, long keys are not truncated, and HDR file names are sanitized with a _N suffix for repeats. (#599, #632, #629)
  • Output errors and standard streams: a write error fails the run and an unwritable --out-file or --failed-keys-file is a fatal startup error (#729, #760, #717); closed stdin, stdout or stderr are reopened on /dev/null, and with stdout closed and no --out-file memtier fails at startup instead of losing results (#736).

Option validation

  • Numeric options: wrapped, overflowing and empty values are rejected, including ports and the --uri database number; leading whitespace and a single + are still accepted, --key-minimum=0 and --reconnect-backoff-factor=0 are accepted. (#699, #721, #748, #761, #782, #668)
  • --uri: percent-decoded credentials and bracketed IPv6 hosts; a host-less --uri warns and uses localhost. (#764, #720)
  • Data options: malformed --data-import files fail fast instead of hanging or crashing (#767); --data-verify and --verify-only exit non-zero on mismatches, missing keys, error replies or no verification replies (#771); __data__ is sized by its command's key under --data-size-pattern=S, __scan_cursor__ is sent as a constant, and __monitor_line@__ placeholders are filled before sending, fixing a cluster hang (#759, #754, #660, #795).

Protocol parsing, libevent and crashes

  • RESP parsing: lengths and counts the parser cannot represent are rejected, and RESP3 pushes, attributes and nested aggregates no longer read past their slot or corrupt the next reply; miss tracking counts RESP3 booleans, and a SCAN reply whose cursor is not a bulk string restarts the iteration. (#832, #846, #859, #882, #881, #854)
  • Multi-channel pub/sub: SUBSCRIBE, PSUBSCRIBE, UNSUBSCRIBE and PUNSUBSCRIBE with several channels complete at their last reply (one per channel) instead of aborting, and their byte and request counts change; UNSUBSCRIBE with no arguments is still counted as one reply. (#875, fixes #515)
  • libevent: TLS error reporting is fixed on libevent 2.1 and 2.2 (a plaintext connection error on 2.2.2-alpha could corrupt memory), libevent 2.1.8's error callback from a failing connect is handled, and a failed bufferevent creation reports an error instead of aborting. (#822, #855, #823)
  • Memcache text parsing: reply keywords are compared only within the line, fixing a one-byte over-read on every GET reply. (#857)
  • Memory safety and crash reports: TCP keepalive and no-delay options are always applied (#541, also in 2.5.2), URI credentials and hostnames are owned correctly (#544, also in 2.5.2), the Unix socket listen queue is retried without blocking the event loop (#258), READONLY frames are fully written (#820), the crash report header avoids localtime() and stdio (#755), and --version and crash-report Git metadata refresh on rebuild (#778).

Build & Configuration Improvements

  • libevent minimum 2.0.11: raised from 2.0.10 for the core, OpenSSL and pthread packages, because 2.0.10 returns monotonic uptime from event_base_gettimeofday_cached() on Linux, which broke deadline-clamped reconnect timing. (#812)
  • Prometheus exporter needs libevent 2.1.4: the documented and configured minimum is corrected from 2.1.1 to 2.1.4 (evhttp_request_set_on_complete_cb() first appeared in 2.1.4-alpha); use --disable-prometheus to build on older 2.1.x. (#823)
  • libtool no longer required, and the spurious LT_INIT: command not found message is gone. (#762)
  • OpenSSL 4.0: the build is fixed. Also in 2.5.2. (#548)
  • Bash completion now installs under --prefix (before, an absolute pkg-config path was used when bash-completion was installed, so non-root prefix installs and make distcheck failed); use --with-bash-completion-dir=DIR or --without-bash-completion-dir. (#765)

Packaging

  • APT repository: Debian 13 (trixie) and Ubuntu 26.04 LTS (resolute) are in the APT release matrix, and Ubuntu armhf/arm64 package builds use a reachable ports mirror. Also in 2.5.1. (#512)
  • Grafana: a Prometheus dashboard at grafana/dashboards/prom-memtier.json for live runs with --prometheus-port. Also in 2.5.1. (#510)
  • Vendored HdrHistogram_c is upgraded to 0.12.0 (unmodified upstream) with SHA-256 checksums verified by make check. (#563)

Developer Tooling & CI

  • Compatibility and sanitizers: libevent 2.1.12, 2.1.13 and 2.2.2-alpha compatibility CI and a Rocky Linux 8 job (libevent 2.1.8), unit tests in the sanitizer workflows, repaired coverage builds, sharded and more reliable CI, and a stricter test harness (#545, #855, #900, #853, #844, #781, #792, #814, #818, #899, #896, #841, #863). Legacy OpenSSL and Debian Bullseye builds and CI on 2.* branches were restored (#540, #550; also in 2.5.2).
  • Opt-in Claude PR review and issue triage workflows, hardened to remove the model's Bash tool grant (#520-#524, #533), and shared Claude/Codex agent instructions (#770).
  • Docs and small items: Docker instructions repository name, a README typo, MANAGER.md (#518, #815, #791); and cheaper per-second latency summaries (#564).

Thanks

New contributors since 2.5.2: @dnlzrgz (#518), @thweetkomputer (#519), @Daniel-Issen (#546), @darth-modder (#552), @sam-obeid (#363), @wanghenshui (#129) and @hamidzr (#815). Also thanks to @zumo64 (#510, 2.5.1) and @collinfunk (#548, 2.5.2).

Full Changelog: 2.5.2...2.6.0