v1.39.0

[romange]

Dragonfly v1.39.0

This release delivers a significant leap in full-text search capability — FT.HYBRID fuses vector similarity with text queries, exact phrase matching with slop arrives, Porter stemming is on by default for TEXT fields, and BM25STD/TFIDF scoring with WITHSCORES support rounds out a deeply improved search engine. Performance improves across the board: PubSub reply batching cuts syscalls by 24–70% and raises throughput up to +12% RPS, INFO REPLICATION becomes fully lock-free, and a new ShardedHashMap replaces the global RCU ChannelStore for pub/sub. Security is tightened with multiple Lua sandbox hardening patches, RESP injection prevention, and PII-safe error logging. Tiered hashes are now mutable. A significant sweep of zombie-key cleanup fixes closes a class of bugs where lazy field expiry across SORT, ZUNIONSTORE, ZINTERSTORE, SHRINK, and hash TTL commands left empty containers in the keyspace.

Highlights

• Added FT.HYBRID combining full-text and vector similarity search with LINEAR and RRF score fusion strategies, capping a major release cycle of search improvements (feat(search): implement FT.HYBRID command with LINEAR/RRF score fusion #7420)
• PubSub, Monitor, and Invalidation replies are now batched per wakeup, delivering up to +12% RPS and 24–70% fewer syscalls at pipeline depths 1–500 (perf(facade): batch V2 control-path replies in ProcessControlMessages #7479)
• Tagged chunk serialization splits large objects across snapshot and replication boundaries so they can be reassembled correctly even with interleaved journal entries (feat(server): tagged chunk write path #7416, feat(server): Implement tagged chunk loading #7070)

Search

• Adds FT.HYBRID combining full-text and VSIM search via LINEAR (ALPHA/BETA weighted sum of normalized scores) and RRF (Reciprocal Rank Fusion) strategies, with HNSW KNN and range-based vector queries, YIELD_SCORE_AS aliasing, LOAD projection, and FT.PROFILE support (feat(search): implement FT.HYBRID command with LINEAR/RRF score fusion #7420)
• Adds exact phrase queries ("hello world") and approximate phrase matching with slop ("..."~N), backed by per-document token positions stored in compressed posting lists; BM25 scoring updated for phrase semantics (feat(search): exact phrase queries with slop and BM25 scoring #7319)
• TEXT fields now apply Porter stemming by default via the vendored Snowball/libstemmer library; per-field NOSTEM, index-level LANGUAGE (default english), and per-document LANGUAGE_FIELD are all supported; FT.INFO surfaces the new flags (feat(search): Porter stemming for TEXT fields with NOSTEM, LANGUAGE, LANGUAGE_FIELD #7295)
• Adds BM25STD (Okapi BM25) relevance scoring engine with per-field TF and document-length tracking for accurate normalization; results can be ranked when a scorer is activated (feat(search): add BM25STD scoring engine with per-field TF tracking #7101)
• Adds TFIDF (TF × ln(N/n)) and TFIDF.DOCNORM (additionally normalized by field length) as new scorer options for FT.SEARCH, FT.PROFILE, and FT.AGGREGATE; scorer dispatch refactored to a function-pointer table (feat(search): add TFIDF and TFIDF.DOCNORM scorers #7200)
• FT.SEARCH now supports WITHSCORES and SCORER parameters; FT.AGGREGATE gains ADDSCORES to inject a __score field enabling downstream GROUPBY/SORTBY/REDUCE on relevance (feat(search): add WITHSCORES, SCORER, ADDSCORES command support for FT.SEARCH/FT.AGGREGATE #7181)
• HNSW vector index memory is now tracked in the search_used memory class and reported in dragonfly_memory_used_bytes via an atomic capacity-based footprint counter (feat: add memory consumption for HNSW #7205)

Performance

• PubSub, Monitor, and Invalidation control-path replies are now accumulated in the send buffer and flushed together per wakeup rather than one sendmsg per message; single-message wakeups still flush immediately [+3% RPS at pipeline=1, +12% at pipeline=10, −24% to −70% syscalls at pipeline=1–500] (perf(facade): batch V2 control-path replies in ProcessControlMessages #7479)
• INFO REPLICATION and replica memory stats are now served lock-free via per-proactor thread-local snapshots updated on every mutation, eliminating global mutex and per-replica shared-lock fan-out (feat: info replication is lock free #7348)
• Replaced the global RCU-based ChannelStore with a ShardedHashMap (16 independent shard locks), removing the pool-wide broadcast on every subscribe/unsubscribe (feat(pubsub): Replace RCU-based ChannelStore with ShardedHashMap #7174)
• NotifyWatchQueue exits in O(1) when a watched key is absent (previously O(N) over all waiters), directly benefiting BLPOP, BZPOPMIN, and XREAD workloads (fix: avoid O(n) scans in NotifyWatchQueue #7225)
• Long-running container iteration commands (LRANGE, SMEMBERS, ZRANGE, HGETALL, etc.) now yield the shard fiber periodically; configurable via --container_iteration_yield_interval_usec (default 500 µs, 0 to disable) (feat(server): Yield fiber periodically during long container iterations #7391)
• Integrates OAHSet as an alternative in-memory set backend selectable via --use_oah_set; threaded through RDB save/load, async deletion, defragmentation, and SHRINK (feat: integrate OAHSet #7246)
• Adds foundational zero-copy GET infrastructure to CompactObj and LargeString: TryBorrow() returns an encoded view with Copy-on-Write semantics; a thread-local pin registry defers buffer deallocation until all readers finish (core: zero-copy GET primitives #7412)
• Connections that occasionally receive large commands now trim parse buffers exceeding 256 KiB at more than 2× actual utilization, preventing permanent oversized memory holds (facade: Trim parsed args if capacity much higher than size #7378)

Replication

• Adds serialization_tagged_chunks flag enabling the RDB serializer to split large entries into tagged chunks with envelope IDs; compression is applied per-blob; a related list-decoding crash during chunked load is also fixed (feat(server): tagged chunk write path #7416)
• Implements tagged chunk loading in the RDB loader with per-stream continuation state, supporting interleaved chunks from different objects and cross-database-boundary reassembly (feat(server): Implement tagged chunk loading #7070)
• Four new Prometheus metrics are now emitted on replica instances at /metrics: dragonfly_master_link_status, dragonfly_master_last_io_seconds_ago, dragonfly_sync_in_progress, and dragonfly_slave_repl_offset (feat: add replication info to /metrics endpoint #7069)

Cluster

• Adds --cluster_coordinator_connect_timeout_ms and --cluster_coordinator_response_timeout_ms flags to control coordinator-to-shard timeouts (previously hard-coded at 3000 ms) (feat(cluster): add flags for coordinator timeouts #7221)
• Cluster config validation now rejects invalid migration configurations at parse time: wrong target masters, duplicate targets, empty or out-of-range slot sets, and overlapping migration ranges (feat: add stricter cluster config validation #7210)

Commands

• CLIENT LIST now supports TYPE <normal|master|replica|slave|pubsub> and ID <id> [id …] filter arguments; outbound replication links are surfaced as TYPE master entries; CLIENT KILL ID on a master-link ID returns a descriptive error directing users to REPLICAOF NO ONE (feat(server): Support CLIENT LIST [TYPE ...] [ID ...] filters #7377)
• DEBUG TRAFFIC now supports per-listener recording (LISTENER <main|memcache|admin>), full Memcache protocol capture and replay (SET/GET/CAS/INCR/GAT/etc. with flags, expiry, and CAS tokens), and a v3 file format; v2 files remain readable (feat: DEBUG TRAFFIC per-listener recording and memcache replay #7192)
• DEBUG TRAFFIC START <path> REPLICA records the replication stream received by a replica from its master; fails with a clear error on standalone/master instances (feat: DEBUG TRAFFIC REPLICA - record replication stream on replicas #7217)
• BF.SCANDUMP iteratively exports a Scalable Bloom Filter in chunks of up to 16 MiB, returning (cursor, data) pairs until cursor=0 signals end-of-stream (feat(bloom): Implement SCANDUMP cmd for SBF #7092)
• BF.LOADCHUNK reconstructs an SBF from chunks produced by BF.SCANDUMP, completing the SCANDUMP/LOADCHUNK round-trip; guards prevent other BF commands from operating on a partially-loaded filter (feat(bloom): Implement LOADCHUNK cmd for SBF #7169)
• BGSAVE now accepts the optional SCHEDULE subcommand (case-insensitive) for Redis client compatibility; concurrent saves are still rejected rather than queued (fix: accept optional SCHEDULE subcommand in BGSAVE #7286)
• dfly_bench gains --json_out_file to write a memtier_benchmark-compatible latency JSON report including per-operation aggregate stats (count, ops/sec, average/min/max, p50/p99/p99.9) and a per-second time-series (Add memtier-compatible latency JSON to dfly_bench #7269)

Security

• The Lua load() built-in is now wrapped to enforce text-only mode ("t"), closing a sandbox escape vector where a malicious script could supply binary Lua bytecode to bypass restrictions (fix(lua): Restrict load() to text-only mode #7376)
• Lua sandbox hardened: rawset, setmetatable, and getmetatable are overridden to block access to _G and all global library tables; guard metatables are attached to prevent replacement or corruption across script executions (fix(lua): Harden sandbox by protecting rawset, setmetatable, and getmetatable #7370)
• Error logs from failed commands no longer include user-supplied PII: arguments are truncated at 128 bytes with a byte-count suffix, at most 31 args are logged, EVAL KEYS/ARGV are omitted, and AUTH/ACL subcommands are fully redacted (fix: prevent PII leaks in FailedCommandToString error logs #7338)
• INFO ACL section added, exposing aggregate registry metrics (acl_num_users, password/glob counts, estimated acl_total_bytes) maintained incrementally without holding the registry write lock (chore: add info acl subcommand and metrics #7149)

Cloud & Storage

• S3 snapshot storage is now available in all builds using helio's native S3 client by default; the AWS SDK backend remains available via WITH_AWS + --s3_use_helio_client=false; builds without WITH_AWS no longer exit on s3:// paths (feat(s3): make AwsS3SnapshotStorage always available; AWS SDK backend optional #7180)
• Azure Blob Storage snapshot backend fully implemented: az:// URIs are correctly routed, OpenWriteFile/OpenReadFile use the cloud::azure API, and cursor-based pagination fixes truncation for large buckets (feat(server): complete AzureSnapshotStorage implementation #7131)
• Hash write commands (HSET, HDEL) now operate on hashes offloaded to tiered (disk-backed) storage: the serialized listpack is read from disk, mutated in-place, and re-stashed with I/O backpressure (feat(tiering): Mutable hash commands #6085)
• S3 endpoint override (e.g. for MinIO) is now passed directly to AwsCredsProvider instead of setting AWS_S3_ENDPOINT in the process environment, eliminating side effects for other processes (feat(aws): pass S3 endpoint directly to AwsCredsProvider #7408)

Bug Fixes

Replication & Persistence

• Fixed slave_repl_offset reporting zero in INFO REPLICATION when replicating from a Redis/Valkey master; offset is now correctly populated from repl_offs_ (fix: slave_repl_offset should be non zero for valkey master #7517)
• Fixed a TOCTOU race where a DflyConn_ fiber could violate the serializer's fiber-identity check during BGSAVE while Lua EVAL scripts ran concurrently (chore: reproduce inline schedule during save #7499)
• Fixed single-shard batch scheduling (ScheduleBatchInShard) allowing a journal DEL entry to execute before the RDB loader finished creating the key on a shard (fix(server): Disable single shard batch optimization #7507)
• Fixed SBF (Scalable Bloom Filter) serialization mismatch where SaveString encoding was read back as raw bytes, crashing Dragonfly in Renamer::FinalizeRename on COPY/DUMP (fix: enable rdb_sbf_chunked by default and fix sbf serialization bug #7475)
• Fixed data loaded via DFLY LOAD not propagating to replicas; the journal ring buffer is now cleared and reconnecting replicas are forced into full sync (fix: replication for data loaded by DFLY LOAD command #6740)
• Fixed slot MOVED redirections becoming corrupted after replica takeover due to REPLCONF CLIENT-ID sending the replica's internal ID instead of its cluster node ID (fix: slot redirection after takeover corrupted #7263)
• Fixed a race in cluster slot migration restart where StartNewSlotMigrations was called before stale slot data was flushed, causing incoming DFLYMIGRATE FLOW writes to be wiped (fix: migration restart flush race #7333)
• Fixed a SIGSEGV during HNSW index replication where per-shard restoration raced with new HSET operations on still-restoring shards; a three-state lifecycle (kProhibit → kRestoring → kBuilding) now defers all graph mutations until all shards complete (feat: deffer HNSW index updates during restoration #7126)

Security & Lua

• Fixed a RESP protocol injection vulnerability where redis.error_reply() / redis.status_reply() strings containing \r\n could inject arbitrary frames into the response stream (fix: strip CRLF from Lua error_reply and status_reply #7332)
• Fixed dragonfly.randstr() accepting unbounded arguments; input is now validated for count (1–32768), size (1–16 MiB), and type (fix(lua): Add input size validation to dragonfly.randstr() #7368)
• Fixed EVAL/EVALSHA crashing with a CHECK abort on negative or non-numeric numkeys arguments (e.g. "-0") instead of returning an error reply (fix(server): reject negative numkeys in EVAL/EVALSHA to avoid CHECK abort #7203)

Search

• Fixed full-text search scoring (BM25, TF-IDF, TFIDF.DOCNORM) using per-shard document stats instead of global aggregates, causing non-deterministic top-K rankings on multi-shard deployments; a two-phase stats-collection pass now precedes the search (fix(search): use global stats for shard-independent text scoring #7250)
• Fixed TEXT field tokenization not honoring backslash escapes (e.g. \., \-, \:) symmetrically at index time and query time, causing escaped-punctuation queries to fail to match (fix(search): honor backslash escapes in TEXT tokenization #7438)
• Fixed a crash during SAVE when FT.SEARCH lazy field expiry left behind an empty zombie hash key; StringMapAccessor's destructor now deletes the key if the hash is empty after access (fix: delete empty hash after FT.SEARCH lazy field expiry #7166)

Commands & Data Structures

• RESTORE now performs deep (entry-by-entry) integrity validation on listpack, intset, quicklist, and stream payloads, preventing crafted payloads with malformed interior entries from causing OOB reads or crashes (fix: deep-validate listpack/intset payloads on RESTORE #7502)
• Fixed DenseSet::SetExpiryTime not setting expiration_used_, causing RandomPairsUnique to skip expiry filtering and return already-expired keys (fix(string_map): mark DenseSet expiration_used in SetExpiryTime #7392)
• Fixed two use-after-free bugs in DenseSet expiry chain walks (ExpireIfNeededInternal, Find2) where deleting a node left node/curr dangling for subsequent iterations (fix(dense_set): avoid use-after-free in expiry chain walks #7393)
• Fixed single-field hashes incorrectly converting to dense_set encoding when the value exceeded max_map_field_len; listpack encoding is now preserved for single-field hashes regardless of value size (fix: single-field hash should use listpack even with large values #7257)
• Fixed an out-of-bounds read in BITCOUNT with BIT index mode when the end index falls exactly on a byte boundary, and incorrect handling of both-negative inverted ranges on short strings (fix: OOB in BITCOUNT and some refactoring #7190)
• Fixed per-type memory accounting underflow when AutoUpdater::Run() processed a value whose object type changed (e.g., SORT STORE overwriting a SET with a LIST) (fix: correct per-type memory accounting when object type changes in AutoUpdater #7142)
• Fixed XADD computing an incorrect memory diff when RecordJournal preempted and triggered evictions before the stream size update; UpdateStreamSize is now called before RecordJournal (fix: memory track before preemption in OpAdd #7235)
• Fixed MEMORY DEFRAGMENT silently ignoring parse errors for the threshold float argument and proceeding with defaults (server: Handle errors in memory cmd parse #7346)
• Fixed MULTI/EXEC ghost connections (addr=0.0.0.0, phase=scheduled) permanently stalling when a socket RST prevented CancelBlocking() from being reached (test(connection): add regression tests for MULTI/EXEC tx-queue stall #7273)
• Fixed a FIN_WAIT_2/CLOSE_WAIT socket leak where connections blocked on BLPOP and similar commands did not observe EOF when the client closed its socket (test(connection): add a fix for FIN_WAIT_2 leak bug with BLPOP #7224)
• Fixed a crash during BGSAVE when a lazily-expired hash field caused an empty container mid-snapshot; DFATAL downgraded to ERROR so the entry is safely skipped (fix(rdb): tolerate transiently empty container during snapshot #7452)
• Fixed a crash in DEBUG OBJHIST/UNIQ-STRS from lazy expiry triggered in an unexpected fiber context, and a second bug where the wrong DB index was used when deleting the resulting zombie key (fix: defer empty-container cleanup in DEBUG OBJHIST/UNIQ-STRS to shard_queue fiber #7241)
• Fixed a bad-input parse error in the RESP server parser logging at ERROR level, which allowed malformed/adversarial clients to flood the error log; downgraded to rate-limited WARNING (facade: Downgrade bad input parse failure to warning #7352)

Zombie Key / Empty Container Cleanup — Fixed a widespread class of bugs where lazy field or member expiry left empty containers as zombie keys, causing DFATAL crashes in subsequent SAVE or incorrect command behavior:

• Empty hash not deleted after: FIELDTTL (fix: delete empty hash after field expiry in FIELDTTL #7137), FIELDEXPIRE (fix: delete empty hash after FIELDEXPIRE lazy expiry #7158), HTTL/HPTTL (fix: delete empty hash after HTTL/HPTTL lazy expiry #7159), DEBUG OBJHIST/UNIQ-STRS (fix: clean up empty containers after DEBUG OBJHIST/UNIQ-STRS #7165)
• Empty set not deleted after: SHRINK compaction (fix: delete empty container after SHRINK compaction #7160), SORT (fix: clean up empty set after SORT lazy expiry #7161), SORT BY pattern (fix: clean up empty set after SORT BY pattern lazy expiry #7162), ZUNIONSTORE (fix: clean up empty set after ZUNIONSTORE lazy expiry #7163), ZINTERSTORE (fix: clean up empty set after ZINTERSTORE lazy expiry #7164)
• Lazy expiry disabled during cluster migration serialization to prevent mid-iteration deletions that caused DFATAL on empty-container saves (fix: disable lazy expiry during migration serialization #7167)

---

What's Changed

• chore(deps): bump github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs from 1.58.1 to 1.65.0 in /contrib/charts/dragonfly by @dependabot[bot] in chore(deps): bump github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs from 1.58.1 to 1.65.0 in /contrib/charts/dragonfly #7088
• chore(deps): bump actions/download-artifact from 7 to 8 in the actions group by @dependabot[bot] in chore(deps): bump actions/download-artifact from 7 to 8 in the actions group #7132
• chore(deps): bump github.com/aws/aws-sdk-go-v2/service/lambda from 1.77.5 to 1.88.5 in /contrib/charts/dragonfly by @dependabot[bot] in chore(deps): bump github.com/aws/aws-sdk-go-v2/service/lambda from 1.77.5 to 1.88.5 in /contrib/charts/dragonfly #7089
• chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.88.2 to 1.97.3 in /contrib/charts/dragonfly by @dependabot[bot] in chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.88.2 to 1.97.3 in /contrib/charts/dragonfly #7090
• fix: empty name doc for search by @BorysTheDev in fix: empty name doc for search #7127
• chore(helm-chart): update to v1.38.0 by @romange in chore(helm-chart): update to v1.38.0 #7135
• fix: memory calculation after GC by @BorysTheDev in fix: memory calculation after GC #7136
• fix: sign helm chart PR commits via GitHub API by @vyavdoshenko in fix: sign helm chart PR commits via GitHub API #7141
• feat(conn): add pipeline backpressure for IoLoopV2 by @glevkovich in feat(conn): add pipeline backpressure for IoLoopV2 #7018
• fix(server): Use BucketSet directly in OnChange by @dranikpg in fix(server): Use BucketSet directly in OnChange #7095
• fix: support LEGACY_GLOG=OFF by @romange in fix: support LEGACY_GLOG=OFF #7139
• fix: delete empty hash after field expiry in FIELDTTL by @vyavdoshenko in fix: delete empty hash after field expiry in FIELDTTL #7137
• fix: correct per-type memory accounting when object type changes in AutoUpdater by @romange in fix: correct per-type memory accounting when object type changes in AutoUpdater #7142
• feat(facade): add connection migration support for IO loop V2 by @glevkovich in feat(facade): add connection migration support for IO loop V2 #7143
• feat(server): Serializer bucket states by @dranikpg in feat(server): Serializer bucket states #7027
• feat(search): add BM25STD scoring engine with per-field TF tracking by @vyavdoshenko in feat(search): add BM25STD scoring engine with per-field TF tracking #7101
• feat(server): complete AzureSnapshotStorage implementation by @romange in feat(server): complete AzureSnapshotStorage implementation #7131
• feat(bloom): Implement SCANDUMP cmd for SBF by @abhijat in feat(bloom): Implement SCANDUMP cmd for SBF #7092
• feat: deffer HNSW index updates during restoration by @BorysTheDev in feat: deffer HNSW index updates during restoration #7126
• chore: split debug info and improve build time on 10% by @BorysTheDev in chore: split debug info and improve build time on 10% #7148
• fix: race condition during flush slots by @BorysTheDev in fix: race condition during flush slots #7145
• fix: delete empty hash after FIELDEXPIRE lazy expiry by @vyavdoshenko in fix: delete empty hash after FIELDEXPIRE lazy expiry #7158
• fix: delete empty hash after HTTL/HPTTL lazy expiry by @vyavdoshenko in fix: delete empty hash after HTTL/HPTTL lazy expiry #7159
• fix(v2): set READ_SOCKET phase and refresh heartbeat for timeouts by @glevkovich in fix(v2): set READ_SOCKET phase and refresh heartbeat for timeouts #7155
• ci: Use Azure mirror for cgroup test to fix apt failures by @glevkovich in ci: Use Azure mirror for cgroup test to fix apt failures #7172
• fix: delete empty container after SHRINK compaction by @vyavdoshenko in fix: delete empty container after SHRINK compaction #7160
• fix: SHRINK command memory accounting causes DCHECK crash by @vyavdoshenko in fix: SHRINK command memory accounting causes DCHECK crash #7173
• refactor: remove duplication logic for HNSW doc deffer processing by @BorysTheDev in refactor: remove duplication logic for HNSW doc deffer processing #7157
• fix: call flushSlots under transaction to avoid race condition by @BorysTheDev in fix: call flushSlots under transaction to avoid race condition #7168
• fix: SHRINK build break from obj_type out of scope after merge by @vyavdoshenko in fix: SHRINK build break from obj_type out of scope after merge #7177
• fix: clean up empty set after SORT lazy expiry by @vyavdoshenko in fix: clean up empty set after SORT lazy expiry #7161
• fix: clean up empty set after SORT BY pattern lazy expiry by @vyavdoshenko in fix: clean up empty set after SORT BY pattern lazy expiry #7162
• fix: disable lazy expiry during migration serialization by @vyavdoshenko in fix: disable lazy expiry during migration serialization #7167
• fix: clean up empty set after ZUNIONSTORE lazy expiry by @vyavdoshenko in fix: clean up empty set after ZUNIONSTORE lazy expiry #7163
• fix: clean up empty set after ZINTERSTORE lazy expiry by @vyavdoshenko in fix: clean up empty set after ZINTERSTORE lazy expiry #7164
• fix: clean up empty containers after DEBUG OBJHIST/UNIQ-STRS by @vyavdoshenko in fix: clean up empty containers after DEBUG OBJHIST/UNIQ-STRS #7165
• feat: add replication info to /metrics endpoint by @EricHayter in feat: add replication info to /metrics endpoint #7069
• fix: delete empty hash after FT.SEARCH lazy field expiry by @vyavdoshenko in fix: delete empty hash after FT.SEARCH lazy field expiry #7166
• fix(server): Reduce mutex scope in snapshot/streamer by @dranikpg in fix(server): Reduce mutex scope in snapshot/streamer #7150
• test: Adjust connection tests for V2 architectural differences by @glevkovich in test: Adjust connection tests for V2 architectural differences #7156
• refactor: move ExecutionState out of ProtocolClient by @kostasrim in refactor: move ExecutionState out of ProtocolClient #7170
• chore: add info acl subcommand and metrics by @kostasrim in chore: add info acl subcommand and metrics #7149
• chore(facade): rename predicates, add IoBuf& params, non-functional improvements by @romange in chore(facade): rename predicates, add IoBuf& params, non-functional improvements #7012
• feat(search): add WITHSCORES, SCORER, ADDSCORES command support for FT.SEARCH/FT.AGGREGATE by @vyavdoshenko in feat(search): add WITHSCORES, SCORER, ADDSCORES command support for FT.SEARCH/FT.AGGREGATE #7181
• docs(transaction.md): Fix Multiple Typos by @RoyTouw77 in docs(transaction.md): Fix Multiple Typos #7194
• fix: OOB in BITCOUNT and some refactoring by @BorysTheDev in fix: OOB in BITCOUNT and some refactoring #7190
• chore: add logs to detect OOM during replication restart process by @BorysTheDev in chore: add logs to detect OOM during replication restart process #7197
• chore(server): Introduce read budget to prep for tagged chunk loading by @abhijat in chore(server): Introduce read budget to prep for tagged chunk loading #7109
• feat: add Apply method to CmdArgParser by @BorysTheDev in feat: add Apply method to CmdArgParser #7204
• fix(server): reject negative numkeys in EVAL/EVALSHA to avoid CHECK abort by @vyavdoshenko in fix(server): reject negative numkeys in EVAL/EVALSHA to avoid CHECK abort #7203
• feat(search): add TFIDF and TFIDF.DOCNORM scorers by @vyavdoshenko in feat(search): add TFIDF and TFIDF.DOCNORM scorers #7200
• feat(bloom): Implement LOADCHUNK cmd for SBF by @abhijat in feat(bloom): Implement LOADCHUNK cmd for SBF #7169
• fix(server): Re-use journal executor by @dranikpg in fix(server): Re-use journal executor #7201
• fix(server): Process empty buckets in snapshot by @dranikpg in fix(server): Process empty buckets in snapshot #7185
• chore(deps): bump github.com/moby/spdystream from 0.5.0 to 0.5.1 in /contrib/charts/dragonfly by @dependabot[bot] in chore(deps): bump github.com/moby/spdystream from 0.5.0 to 0.5.1 in /contrib/charts/dragonfly #7175
• chore(server): Extended SerializerBase test by @dranikpg in chore(server): Extended SerializerBase test #7179
• test: add logging to bullmq tests by @vyavdoshenko in test: add logging to bullmq tests #7209
• chore(deps): bump github.com/jackc/pgx/v5 from 5.7.6 to 5.9.2 in /contrib/charts/dragonfly by @dependabot[bot] in chore(deps): bump github.com/jackc/pgx/v5 from 5.7.6 to 5.9.2 in /contrib/charts/dragonfly #7207
• feat: add memory consumption for HNSW by @BorysTheDev in feat: add memory consumption for HNSW #7205
• fix(server): Don't use item queue for inlined load by @dranikpg in fix(server): Don't use item queue for inlined load #7211
• fix: replication for data loaded by DFLY LOAD command by @BorysTheDev in fix: replication for data loaded by DFLY LOAD command #6740
• build(deps): bump filippo.io/edwards25519 from 1.1.0 to 1.1.1 in /contrib/charts/dragonfly by @dependabot[bot] in build(deps): bump filippo.io/edwards25519 from 1.1.0 to 1.1.1 in /contrib/charts/dragonfly #6678
• tests: Update pytest by @abhijat in tests: Update pytest #7212
• feat: new methods in CmdArgParser by @BorysTheDev in feat: new methods in CmdArgParser #7208
• feat: add stricter cluster config validation by @BorysTheDev in feat: add stricter cluster config validation #7210
• feat: DEBUG TRAFFIC per-listener recording and memcache replay by @vyavdoshenko in feat: DEBUG TRAFFIC per-listener recording and memcache replay #7192
• feat: DEBUG TRAFFIC REPLICA - record replication stream on replicas by @vyavdoshenko in feat: DEBUG TRAFFIC REPLICA - record replication stream on replicas #7217
• feat(core): Add ShardedHashMap — sharded concurrent hash map by @mkaruza in feat(core): Add ShardedHashMap — sharded concurrent hash map #7196
• fix: avoid O(n) scans in NotifyWatchQueue by @kostasrim in fix: avoid O(n) scans in NotifyWatchQueue #7225
• test(connection): add a fix for FIN_WAIT_2 leak bug with BLPOP by @romange in test(connection): add a fix for FIN_WAIT_2 leak bug with BLPOP #7224
• tests: Wait for monitor set up before firing commands by @abhijat in tests: Wait for monitor set up before firing commands #7226
• fix(search): propagate FLAT VECTOR_RANGE score alias to FT.SEARCH reply by @vyavdoshenko in fix(search): propagate FLAT VECTOR_RANGE score alias to FT.SEARCH reply #7222
• chore(server) server: Add mem buf controller by @abhijat in chore(server) server: Add mem buf controller #7198
• perf(facade): defer V2 ReplyBatch flush to reduce syscalls by @glevkovich in perf(facade): defer V2 ReplyBatch flush to reduce syscalls #7213
• test: add more HNSW tests by @BorysTheDev in test: add more HNSW tests #7216
• feat(pubsub): Replace RCU-based ChannelStore with ShardedHashMap by @mkaruza in feat(pubsub): Replace RCU-based ChannelStore with ShardedHashMap #7174
• refactor: drop extra metadata for HNSW replication by @BorysTheDev in refactor: drop extra metadata for HNSW replication #7215
• build(deps): update meta-memcache requirement from >=2 to >=2.2.0 in /tests/dragonfly by @dependabot[bot] in build(deps): update meta-memcache requirement from >=2 to >=2.2.0 in /tests/dragonfly #7228
• build(deps): update pyyaml requirement from >=6.0 to >=6.0.3 in /tests/dragonfly by @dependabot[bot] in build(deps): update pyyaml requirement from >=6.0 to >=6.0.3 in /tests/dragonfly #7231
• build(deps): update pytest-icdiff requirement from >=0.8 to >=0.9 in /tests/dragonfly by @dependabot[bot] in build(deps): update pytest-icdiff requirement from >=0.8 to >=0.9 in /tests/dragonfly #7230
• build(deps): update packaging requirement from >=23.1 to >=26.2 in /tests/dragonfly by @dependabot[bot] in build(deps): update packaging requirement from >=23.1 to >=26.2 in /tests/dragonfly #7229
• feat: add lacked functionality for OAHSet by @BorysTheDev in feat: add lacked functionality for OAHSet #7233
• feat(core): Introduce zero-alloc StreamNodeObj tagged pointer by @mkaruza in feat(core): Introduce zero-alloc StreamNodeObj tagged pointer #7104
• feat: support ~ optional match operator in query language by @vyavdoshenko in feat: support ~ optional match operator in query language #7227
• feat(s3): make AwsS3SnapshotStorage always available; AWS SDK backend optional by @romange in feat(s3): make AwsS3SnapshotStorage always available; AWS SDK backend optional #7180
• fix: memory track before preemption in OpAdd by @kostasrim in fix: memory track before preemption in OpAdd #7235
• feat(server): Implement tagged chunk loading by @abhijat in feat(server): Implement tagged chunk loading #7070
• fix: handle incompressible data in HuffmanCheckTask by @shitaoxai in fix: handle incompressible data in HuffmanCheckTask #7251
• tests: Wait for slot flush propagation by @abhijat in tests: Wait for slot flush propagation #7243
• chore: remove a temporary barrier on replica executor side by @romange in chore: remove a temporary barrier on replica executor side #7214
• feat(tools): add AI-powered release notes generator by @vyavdoshenko in feat(tools): add AI-powered release notes generator #7252
• fix: defer empty-container cleanup in DEBUG OBJHIST/UNIQ-STRS to shard_queue fiber by @vyavdoshenko in fix: defer empty-container cleanup in DEBUG OBJHIST/UNIQ-STRS to shard_queue fiber #7241
• fix: single-field hash should use listpack even with large values by @romange in fix: single-field hash should use listpack even with large values #7257
• feat(tools): add Copilot backend, per-commit cache, and async refactor to release notes generator by @romange in feat(tools): add Copilot backend, per-commit cache, and async refactor to release notes generator #7264
• fix(tests): Improve shutdown save test by @dranikpg in fix(tests): Improve shutdown save test #7237
• feat(tools): extend release notes generator with announce target by @romange in feat(tools): extend release notes generator with announce target #7268
• chore: adjust code for LEGACY_GLOG=OFF build by @romange in chore: adjust code for LEGACY_GLOG=OFF build #7154
• Add memtier-compatible latency JSON to dfly_bench by @romange in Add memtier-compatible latency JSON to dfly_bench #7269
• fix: accept optional SCHEDULE subcommand in BGSAVE by @vyavdoshenko in fix: accept optional SCHEDULE subcommand in BGSAVE #7286
• ci: skip flaky BullMQ QueueEvents.run() race test by @vyavdoshenko in ci: skip flaky BullMQ QueueEvents.run() race test #7292
• CI: match mastodon tests to upstream by @abhijat in CI: match mastodon tests to upstream #7307
• chore: pull latest helio by @kostasrim in chore: pull latest helio #7310
• fix(ci): pin helm version to v3.21.0 to match golden files by @vyavdoshenko in fix(ci): pin helm version to v3.21.0 to match golden files #7311
• feat(search): Porter stemming for TEXT fields with NOSTEM, LANGUAGE, LANGUAGE_FIELD by @vyavdoshenko in feat(search): Porter stemming for TEXT fields with NOSTEM, LANGUAGE, LANGUAGE_FIELD #7295
• fix(search): guard RebuildForGroup against freed DocIds on lazy key expiry by @vyavdoshenko in fix(search): guard RebuildForGroup against freed DocIds on lazy key expiry #7313
• test(connection): add regression tests for MULTI/EXEC tx-queue stall by @romange in test(connection): add regression tests for MULTI/EXEC tx-queue stall #7273
• feat(tiering): Mutable hash commands by @dranikpg in feat(tiering): Mutable hash commands #6085
• fix(server): use LOG_EVERY_T for OOM error logging by @bitborne in fix(server): use LOG_EVERY_T for OOM error logging #7220
• fix(server): Reconcile bucket set by @dranikpg in fix(server): Reconcile bucket set #7260
• fix(ci): install libheif codec plugins for HEIC/AV1 support in Mastodon tests by @vyavdoshenko in fix(ci): install libheif codec plugins for HEIC/AV1 support in Mastodon tests #7315
• fix(server): Fix iterator invalidation by @dranikpg in fix(server): Fix iterator invalidation #7316
• chore: fix test shutdown bug by @romange in chore: fix test shutdown bug #7317
• build(deps): bump the actions group across 1 directory with 2 updates by @dependabot[bot] in build(deps): bump the actions group across 1 directory with 2 updates #7193
• feat(cluster): add flags for coordinator timeouts by @bitborne in feat(cluster): add flags for coordinator timeouts #7221
• refactor: store hnsw metadata together with index by @BorysTheDev in refactor: store hnsw metadata together with index #7242
• fix: strip CRLF from Lua error_reply and status_reply by @vyavdoshenko in fix: strip CRLF from Lua error_reply and status_reply #7332
• fix(tests): Transient 502 network errors crash test suite due to missing retries in urllib (Transient 502 network errors crash test suite due to missing retries in urllib downloads #7238) by @paarth-k2002 in fix(tests): Transient 502 network errors crash test suite due to missing retries in urllib (#7238) #7314
• CI: allow arm in repeat-test by @abhijat in CI: allow arm in repeat-test #7334
• refactor: add method CompactObject::SetMemberTime by @BorysTheDev in refactor: add method CompactObject::SetMemberTime #7335
• fix: slot redirection after takeover corrupted by @kostasrim in fix: slot redirection after takeover corrupted #7263
• fix: migration restart flush race by @kostasrim in fix: migration restart flush race #7333
• fix(test): Disable cooling for pytest with tiering and cluster by @mkaruza in fix(test): Disable cooling for pytest with tiering and cluster #7330
• feat(server): Make DbSlice consumer an interface by @dranikpg in feat(server): Make DbSlice consumer an interface #7253
• server: Handle errors in memory cmd parse by @abhijat in server: Handle errors in memory cmd parse #7346
• feat(search): exact phrase queries with slop and BM25 scoring by @vyavdoshenko in feat(search): exact phrase queries with slop and BM25 scoring #7319
• facade: Downgrade bad input parse failure to warning by @abhijat in facade: Downgrade bad input parse failure to warning #7352
• fix: prevent PII leaks in FailedCommandToString error logs by @vyavdoshenko in fix: prevent PII leaks in FailedCommandToString error logs #7338
• fix(search): use global stats for shard-independent text scoring by @vyavdoshenko in fix(search): use global stats for shard-independent text scoring #7250
• core: split LargeString out of RobjWrapper by @romange in core: split LargeString out of RobjWrapper #7361
• feat(server): Journal omits by @dranikpg in feat(server): Journal omits #7060
• test: fix flaky cluster migration tiering test by @romange in test: fix flaky cluster migration tiering test #7367
• core: replace ROBJ_TAG with per-collection tags, drop RobjWrapper::type_ by @romange in core: replace ROBJ_TAG with per-collection tags, drop RobjWrapper::type_ #7364
• fix(lua): Add input size validation to dragonfly.randstr() by @mkaruza in fix(lua): Add input size validation to dragonfly.randstr() #7368
• fix(lua): Harden sandbox by protecting rawset, setmetatable, and getmetatable by @mkaruza in fix(lua): Harden sandbox by protecting rawset, setmetatable, and getmetatable #7370
• chore: more logs for replication tests by @BorysTheDev in chore: more logs for replication tests #7353
• fix(lua): Restrict load() to text-only mode by @mkaruza in fix(lua): Restrict load() to text-only mode #7376
• fix(server): Enable journal omits only under flag by @dranikpg in fix(server): Enable journal omits only under flag #7371
• chore: release action improvements by @romange in chore: release action improvements #7256
• fix(search): suppress automove double-use warnings in parser.y by @vyavdoshenko in fix(search): suppress automove double-use warnings in parser.y #7383
• facade: Trim parsed args if capacity much higher than size by @abhijat in facade: Trim parsed args if capacity much higher than size #7378
• perf(facade): use CycleClock for ReplyBuilder send timing by @romange in perf(facade): use CycleClock for ReplyBuilder send timing #7388
• perf(server): reduce ProactorBase::GetMonotonicTimeNs callers by @romange in perf(server): reduce ProactorBase::GetMonotonicTimeNs callers #7387
• fix(dense_set): avoid use-after-free in expiry chain walks by @vyavdoshenko in fix(dense_set): avoid use-after-free in expiry chain walks #7393
• perf(server): use CycleClock for streamer/disk-storage timing by @romange in perf(server): use CycleClock for streamer/disk-storage timing #7394
• fix(string_map): mark DenseSet expiration_used in SetExpiryTime by @vyavdoshenko in fix(string_map): mark DenseSet expiration_used in SetExpiryTime #7392
• Increase test_big_huge_streaming_restart timeout by @dranikpg in Increase test_big_huge_streaming_restart timeout #7384
• ci: build with LEGACY_GLOG=OFF by @romange in ci: build with LEGACY_GLOG=OFF #7395
• perf(cluster): use CycleClock for incoming migration throttle timing by @romange in perf(cluster): use CycleClock for incoming migration throttle timing #7398
• facade: Fix comment floor size for arg trim by @abhijat in facade: Fix comment floor size for arg trim #7402
• Remove SerializedMap, store listpack directly in tiered value by @dranikpg in Remove SerializedMap, store listpack directly in tiered value #7374
• feat: integrate OAHSet by @BorysTheDev in feat: integrate OAHSet #7246
• fix(server): run RANDOMKEY scan on regular fiber to allow preempt by @vyavdoshenko in fix(server): run RANDOMKEY scan on regular fiber to allow preempt #7405
• refactor(test): collapse duplicate tiering migration tests into shared helper by @romange in refactor(test): collapse duplicate tiering migration tests into shared helper #7409
• fix(tests): disable omit assert by @dranikpg in fix(tests): disable omit assert #7413
• feat(facade): bound IoLoopV2 dispatch_q_ quota to prevent starvation by @glevkovich in feat(facade): bound IoLoopV2 dispatch_q_ quota to prevent starvation #7234
• feat(aws): pass S3 endpoint directly to AwsCredsProvider by @romange in feat(aws): pass S3 endpoint directly to AwsCredsProvider #7408
• feat(compact_object): extend Huffman encoding up to 16KB with varint header by @romange in feat(compact_object): extend Huffman encoding up to 16KB with varint header #7406
• chore(server): Make listpack bounds configurable by @dranikpg in chore(server): Make listpack bounds configurable #7407
• feat(server): Support CLIENT LIST [TYPE ...] [ID ...] filters by @vyavdoshenko in feat(server): Support CLIENT LIST [TYPE ...] [ID ...] filters #7377
• chore: do not unwrap parsed expression in tests by @kostasrim in chore: do not unwrap parsed expression in tests #7380
• feat(facade): add per-protocol IoLoopV2 flags for Memcache and RESP by @glevkovich in feat(facade): add per-protocol IoLoopV2 flags for Memcache and RESP #7424
• feat: integrate SIMD into OAHSet by @BorysTheDev in feat: integrate SIMD into OAHSet #7306
• fix(test): Remove stale keyspace assertion in _run_tiering_migration by @mkaruza in fix(test): Remove stale keyspace assertion in _run_tiering_migration #7427
• core: zero-copy GET primitives by @romange in core: zero-copy GET primitives #7412
• refactor: ReplicaInfo into class by @BorysTheDev in refactor: ReplicaInfo into class #7415
• feat(server): Yield fiber periodically during long container iterations by @mkaruza in feat(server): Yield fiber periodically during long container iterations #7391
• test: add long HNSW tests by @BorysTheDev in test: add long HNSW tests #7418
• fix(search): preserve decimal GEO radii by @fallintoplace in fix(search): preserve decimal GEO radii #7422
• chore: remove test_tls_when_read_write_is_interleaved by @kostasrim in chore: remove test_tls_when_read_write_is_interleaved #7429
• docs: update pub-sub.md to reflect ShardedHashMap architecture by @Copilot in docs: update pub-sub.md to reflect ShardedHashMap architecture #7239
• chore: factor out boilerplate for tests by @kostasrim in chore: factor out boilerplate for tests #7419
• test: relax ClientPause GET timing budget for sanitizer builds by @vyavdoshenko in test: relax ClientPause GET timing budget for sanitizer builds #7445
• fix(search): honor backslash escapes in TEXT tokenization by @vyavdoshenko in fix(search): honor backslash escapes in TEXT tokenization #7438
• fix(search): preserve empty TAG/TEXT values in FT.SEARCH RETURN by @vyavdoshenko in fix(search): preserve empty TAG/TEXT values in FT.SEARCH RETURN #7434
• test(search): wait for initial indexing before FT.SEARCH by @vyavdoshenko in test(search): wait for initial indexing before FT.SEARCH #7444
• chore(server): Transaction inlining comments by @dranikpg in chore(server): Transaction inlining comments #7441
• docs: rewrite transaction.md to align with current codebase by @romange in docs: rewrite transaction.md to align with current codebase #7448
• feat(server): tagged chunk write path by @abhijat in feat(server): tagged chunk write path #7416
• chore: dfly version clean ups by @romange in chore: dfly version clean ups #7442
• perf: zero-copy GET infrastructure (common, core, facade) by @romange in perf: zero-copy GET infrastructure (common, core, facade) #7433
• fix(rdb): tolerate transiently empty container during snapshot by @vyavdoshenko in fix(rdb): tolerate transiently empty container during snapshot #7452
• fix(server): Cancel transactions by @dranikpg in fix(server): Cancel transactions #7399
• fix(ci): correct docker health check condition by @fallintoplace in fix(ci): correct docker health check condition #7423
• perf(facade): defer pipeline flushes in IoLoopV2 by @glevkovich in perf(facade): defer pipeline flushes in IoLoopV2 #7437
• test(search): wait for indexing before searching populated index by @vyavdoshenko in test(search): wait for indexing before searching populated index #7464
• Add cluster design doc by @BorysTheDev in Add cluster design doc #7435
• feat(search): support w'...' glob wildcards on text and tag fields by @vyavdoshenko in feat(search): support w'...' glob wildcards on text and tag fields #7456
• ci: parallelize ctest runs with -j$(nproc) by @vyavdoshenko in ci: parallelize ctest runs with -j$(nproc) #7440
• feat(search): implement FT.HYBRID command with LINEAR/RRF score fusion by @vyavdoshenko in feat(search): implement FT.HYBRID command with LINEAR/RRF score fusion #7420
• fix(search): accept phrases and affixes in parenthesized field conditions by @vyavdoshenko in fix(search): accept phrases and affixes in parenthesized field conditions #7455
• refactor(search): factor shared query-grammar atoms into term_atom by @vyavdoshenko in refactor(search): factor shared query-grammar atoms into term_atom #7467
• chore: disable default settings that cause regression in CI by @romange in chore: disable default settings that cause regression in CI #7465
• tests: Fix test_denyoom_commands by removing RSS factor by @abhijat in tests: Fix test_denyoom_commands by removing RSS factor #7446
• Fix bigvalue cancel by @dranikpg in Fix bigvalue cancel #7470
• fix: HNSW index building bug on replica by @BorysTheDev in fix: HNSW index building bug on replica #7453
• perf(facade): batch V2 control-path replies in ProcessControlMessages by @glevkovich in perf(facade): batch V2 control-path replies in ProcessControlMessages #7479
• readme(readme): add user conference link by @worldsoup in readme(readme): add user conference link #7483
• fix(search): parse RETURN/LOAD count as tokens, not field-specs by @vyavdoshenko in fix(search): parse RETURN/LOAD count as tokens, not field-specs #7482
• chore(server): Small snapshot updates by @dranikpg in chore(server): Small snapshot updates #7480
• fix(json): surface JSON.DEBUG arg parse errors instead of aborting by @vyavdoshenko in fix(json): surface JSON.DEBUG arg parse errors instead of aborting #7493
• feat: info replication is lock free by @BorysTheDev in feat: info replication is lock free #7348
• ci: run C++ unit tests serially (drop ctest -j) by @vyavdoshenko in ci: run C++ unit tests serially (drop ctest -j) #7495
• chore: drop extra drakarys logs by @BorysTheDev in chore: drop extra drakarys logs #7490
• feat(search): accept bare identifiers in FT.CREATE schema on JSON by @vyavdoshenko in feat(search): accept bare identifiers in FT.CREATE schema on JSON #7488
• fix: enable rdb_sbf_chunked by default and fix sbf serialization bug by @romange in fix: enable rdb_sbf_chunked by default and fix sbf serialization bug #7475
• chore: reduce log level for Drakarys by @BorysTheDev in chore: reduce log level for Drakarys #7497
• docs: add HNSW replication doc by @BorysTheDev in docs: add HNSW replication doc #7219
• fix: deep-validate listpack/intset payloads on RESTORE by @vyavdoshenko in fix: deep-validate listpack/intset payloads on RESTORE #7502
• fix(server): Propagate error on bigvalue cancel by @dranikpg in fix(server): Propagate error on bigvalue cancel #7486
• refactor: drop extra dispatch for info replica section by @BorysTheDev in refactor: drop extra dispatch for info replica section #7503
• fix(server): Disable single shard batch optimization by @abhijat in fix(server): Disable single shard batch optimization #7507
• chore: reproduce inline schedule during save by @kostasrim in chore: reproduce inline schedule during save #7499
• refactor: add more SIMD into OAHSet by @BorysTheDev in refactor: add more SIMD into OAHSet #7478
• feat: improve HSETEX compatibility with Redis by @BorysTheDev in feat: improve HSETEX compatibility with Redis #7510
• CI/build: switch from glog to absl by @abhijat in CI/build: switch from glog to absl #7514
• server: Remove redundant ScheduleContext struct by @abhijat in server: Remove redundant ScheduleContext struct #7516
• fix: slave_repl_offset should be non zero for valkey master by @kostasrim in fix: slave_repl_offset should be non zero for valkey master #7517
• fix: increase timeout for test_randomkey_during_bgsave by @kostasrim in fix: increase timeout for test_randomkey_during_bgsave #7519
• test(fuzz): full command coverage and a valid, repaired RESP seed corpus by @vyavdoshenko in test(fuzz): full command coverage and a valid, repaired RESP seed corpus #7511
• fix: batched backpressure by @dranikpg in fix: batched backpressure #7515

Huge thanks to all the contributors! ❤️

New Contributors

• @RoyTouw77 made their first contribution in docs(transaction.md): Fix Multiple Typos #7194
• @shitaoxai made their first contribution in fix: handle incompressible data in HuffmanCheckTask #7251
• @bitborne made their first contribution in fix(server): use LOG_EVERY_T for OOM error logging #7220
• @paarth-k2002 made their first contribution in fix(tests): Transient 502 network errors crash test suite due to missing retries in urllib (#7238) #7314
• @fallintoplace made their first contribution in fix(search): preserve decimal GEO radii #7422

Full Changelog: v1.38.0...v1.39.0

---

This discussion was created from the release v1.39.0.