v1.24.0

Dragonfly v1.24.0

It's a maintenance release with mostly performance and bug fixes for edge cases.

Some prominent changes include:

1. feat: add HEXPIRE and FIELDEXPIRE by @NegatioN #3780
2. fix BITPOS command responses by @Diskein #3910
3. add allcommands/nocommands to ACL rules by @Lakshyadevelops #3783

What's Changed

• fix: Fix flaky test test_acl_revoke_pub_sub_while_subscribed by @chakaz in #3768
• chore: Forbid replicating a replica by @chakaz in #3779
• fix: GETRANGE params validation by @BorysTheDev in #3781
• fix: bitop do not add dst key if result is empty by @kostasrim in #3751
• chore(acl_family): add allcomands and nocommands by @Lakshyadevelops in #3783
• chore: remove goto statements by @kostasrim in #3791
• chore: ScheduleInternal refactoring by @romange in #3794
• test: update test_noreply_pipeline to prevent false fail by @BorysTheDev in #3801
• fix: bitcount invalid range by @BorysTheDev in #3792
• fix(generic_family): Update indexes in the RESTORE and RENAME commands by @BagritsevichStepan in #3803
• chore: cleanup not used opcodes in replication by @adiholden in #3804
• chore: tune TieredStorageTest.MemoryPressure by @romange in #3805
• fix: make sure dfly_bench reliably connects by @romange in #3802
• chore: reformat listpack according to valkey 8 by @romange in #3810
• fix: empty strings and Nan in the sort command by @BorysTheDev in #3809
• chore: refactor XRead by @dranikpg in #3775
• chore: singlehop XREAD by @dranikpg in #3817
• chore: introduce an optimized integer compare algorithm for lists by @romange in #3813
• chore: add a benchmark comparing lpStringToInt64 to SimpleAtoi by @romange in #3815
• feat(rdb_load): add support for loading huge sets by @andydunstall in #3807
• chore: now it's not needed to allocate quicklistIter on heap by @romange in #3814
• fix(rdb_load): fix loading huge hmaps with ttl by @andydunstall in #3825
• chore: allow config set notify_keyspace_events by @kostasrim in #3790
• chore: do not close connections at the end of pytest by @kostasrim in #3811
• fix(rdb_load): fix appending to an expired key by @andydunstall in #3829
• chore: add logs to test_acl_cat_commands_multi_exec_squash by @kostasrim in #3826
• chore: optimize element removal for integer lists by @romange in #3827
• refactor: remove CmdArgParser::ToUpper() by @BorysTheDev in #3831
• chore: add pending_ops to disk stats by @romange in #3837
• feat(rdb_load): add support for loading huge hmaps and zsets by @andydunstall in #3823
• add dense_set.SetExpiryTime in preparation for fieldexpire by @NegatioN in #3780
• test: more stabe test_bgsave_and_save by @adiholden in #3843
• chore: WrapSds from family_utils.h by @dranikpg in #3818
• chore: add defrag logic for sets by @kostasrim in #3833
• chore: extend RobjWrapper::sz_ to 2^56 by @romange in #3849
• feat(rdb_load): add support for loading huge lists by @andydunstall in #3850
• fix(rdb_load): fix partial reads dropping elements by @andydunstall in #3853
• refactor: refactor fakeredis tests by @cunla in #3852
• chore: simplify CloneBatch code by @romange in #3862
• feat: add HEXPIRE and FIELDEXPIRE by @NegatioN in #3842
• chore: improve performance of ClearInternal by @romange in #3863
• chore: Remove DenseSet::AddOrFindDense and AddSds by @romange in #3864
• chore: Implement AddMany method by @romange in #3866
• chore: ClearInternal now can clear partially by @romange in #3867
• feat(rdb_load): add support for loading huge streams by @andydunstall in #3855
• chore: Refactor AddMany by @romange in #3869
• chore: run memory decommit after snapshot load/save by @kostasrim in #3828
• chore: reduce usage of ToUpper by @romange in #3874
• opt: Optimize AllocationTracker to be efficient when enabled by @chakaz in #3875
• chore: allow rdb version 12 by @kostasrim in #3860
• chore: lock keys for optimistic transactions by @kostasrim in #3865
• fix: Do not publish to connections without context by @chakaz in #3873
• fix: Do not kill Dragonfly on failed DFLY LOAD by @chakaz in #3892
• chore: collection size by @dranikpg in #3844
• chore: skip flaky test_noreply_pipeline by @kostasrim in #3903
• fix(cluster): fix unknown migration error by @andydunstall in #3899
• chrore: parse cgroup v2 by @kostasrim in #3857
• feat(search): STOPWORDS by @dranikpg in #3851
• chore: schedule chains by @romange in #3819
• fix!: fix BITPOS command responses (#3893) by @Diskein in #3910
• fix: prevents Dragonfly from blocking in epoll during snapshotting by @romange in #3911
• fix: macos build by @romange in #3912
• chore(search): Rax TreeMap by @dranikpg in #3909
• chore: Add --allocator_tracker for default tracking by @chakaz in #3901
• chore: relax assertion in test_noreply_pipeline by @kostasrim in #3908
• feat(search): Prefix search* by @dranikpg in #3913
• fix: pre-commit ci workflow by @kostasrim in #3917
• fix: Support replicating Valkey and Redis 7.2 by @chakaz in #3927
• feat(server): use listpack node encoding for list by @adiholden in #3914
• fix: Fix test_flushall_in_full_sync by @chakaz in #3929
• fix: dragonfly_connection should only access the original reply_builder by @romange in #3924
• chore(search): Fix deprecated functions by @dranikpg in #3933

New Contributors

• @Lakshyadevelops made their first contribution in #3783
• @NegatioN made their first contribution in #3780
• @Diskein made their first contribution in #3910

Huge thanks to all the contributors! ❤️

Full Changelog: v1.23.0...v1.24.0

v1.23.2

This is a patch release.
fix: rdb load fix loading plain lists 4959bef

full change log: v1.23.1...v1.23.2

v1.23.1

Dragonfly v1.23.1

This is a patch release.
Update urgency - low.

fix: use less memory when loading huge values: #3829, #3825, #3807
fix: crash in stream code + performance improvements: #3775
fix: crash when using RENAME with search indexes: #3803

Full Changelog: v1.23.0...v1.23.1

v1.23.0

Dragonfly v1.23.0

Some prominent changes include:

• Fix container healthchecks for both privileged and non-privileged environments #3644
• Fix FT.AGGREGATE command (#3657 #3660 )
• Fix regression with the S3 access #3717
• Added ZRANGESTORE command #3757
• introducerss_oom_deny_ratio flag (#3702) - now Dragonfly will reject write commands if its RSS memory usage
  crosses maxmemory*rss_oom_deny_ratio. The default value of this flag is set to 1.25.
• Support for "CLIENT ID" and "CLIENT SETINFO" commands (#3673 and #3672)
• Better memory tracking for JSON objects (#3641)

What's Changed

• fix(docker/healthcheck): run netstat port retreival command as dfly by @Pothulapati in #3647
• test: remove redundant tests by @cunla in #3642
• chore: allow disabling io_uring registered buffers by @romange in #3650
• CmdArgParser improvement by @BorysTheDev in #3633
• fix: deadlock in the cluster migration process by @BorysTheDev in #3653
• fix: fix multi mget exec error message by @BorysTheDev in #3662
• test: fix MC test_expiration by @BorysTheDev in #3663
• chore: add some community suggestions to README by @Niennienzz in #3664
• fix: recursive calls in the allocation tracker by @romange in #3665
• chore: cosmetic changes around Snapshot functions by @romange in #3652
• fix(search_family): Fix FT.AGGREGATE GROUPBY option by @BagritsevichStepan in #3657
• chore: Add CLIENT ID command by @chakaz in #3672
• deprecate RecordsPopper and serialize channel records during push by @romange in #3667
• feat(server): Support CLIENT SETINFO by @chakaz in #3673
• server logs: change script error to warning by @adiholden in #3670
• fix(search_family): Fix LOAD option behavior in the FT.AGGREGATE command by @BagritsevichStepan in #3660
• test: unskip cluster tests and add debug info by @BorysTheDev in #3681
• chore: adjust RdbChannel sizes by @romange in #3676
• chore: add a script that parses allocator tracking logs by @romange in #3687
• chore: limit number of descriptors in the exec map by @romange in #3688
• fix(server): fix replication master deadlock on cancelation flow by @adiholden in #3686
• fix: join for cancel incoming migration by @BorysTheDev in #3692
• fix(json_family): Fix the JSON.SET bug if the path is in legacy mode and is not the root by @BagritsevichStepan in #3693
• fix: test_acl_revoke_pub_sub_while_subscribed by @kostasrim in #3680
• test: fix test_cluster_replication_migration by @BorysTheDev in #3699
• fix: test_replicaof_reject_on_load assert failure by @kostasrim in #3697
• fix(json_family) Add NOESCAPE option to the JSON.GET command by @BagritsevichStepan in #3685
• chore: pull latest helio by @kostasrim in #3700
• feat(core): Support RISCV RVV by @howjmay in #3655
• fix: allow parsing extra spaces on acl files by @kostasrim in #3703
• Update dflycluster slot-migration-status reply by @BorysTheDev in #3707
• chore: add clone benchmark by @romange in #3709
• fix: add default value has_mc_flag field by @BorysTheDev in #3710
• chore: symbolize stack traces in tests upon crash by @romange in #3714
• fix: fix s3 load snapshot by @andydunstall in #3717
• fix(cluster): Do not Pause() replication / migrations by @chakaz in #3716
• chore: add log info on failed commands by @kostasrim in #3694
• chore: enable ReplyGuard in ReplyBuilder2 by @kostasrim in #3705
• tests: fix and enable s3 snapshot test by @andydunstall in #3720
• fix(json_family): Fix JSON.ARRPOP command in legacy mode should not return WRONGTYPE error by @BagritsevichStepan in #3683
• fix(json_family): Separate double and int values during the comparison of the JSON objects by @BagritsevichStepan in #3711
• fix: Fix test test_client_pause_with_replica by @chakaz in #3729
• fix: Fix cluster_mgr.py by @chakaz in #3730
• fix: Fix test_take_over_seeder by @chakaz in #3733
• chore: Remove ReqSerializer by @dranikpg in #3721
• fix(json_family): Fix out of bound ranges for the JSON.ARR* commands by @BagritsevichStepan in #3712
• fix: properly track json memory usage by @kostasrim in #3641
• test: add tests for replication by @BorysTheDev in #3734
• test: unskip test for debugging purpose by @BorysTheDev in #3738
• refactor: use CmdArgParser for XGROUP command by @BorysTheDev in #3739
• test: add test for replication deadlock on replication timeout by @adiholden in #3691
• chore: Switch ports for cluster_mgr_test.py by @chakaz in #3741
• chore: change log level to warning for empty keys by @kostasrim in #3722
• chore: add SetReplies by @dranikpg in #3727
• chore: introduce a Clone function for the dense set by @romange in #3740
• chore: add missing await in test_take_over_seeder by @kostasrim in #3744
• fix: allow the healthcheck run in non-privileged containers as well by @romange in #3731
• fix: add value range check for SETBIT command by @BorysTheDev in #3750
• fix: zset store conclude transaction on error by @andydunstall in #3755
• chore: refactor a lambda function into a named one by @romange in #3753
• chore: fix return on bad status by @dranikpg in #3758
• feat(helm): add hostNetwork, topologySpreadConstraint and clusterIP su… by @katapultcloud in #3389
• feat(server): introduce oom_deny_commands flag by @adiholden in #3718
• feat(server): introduce rss oom limit by @adiholden in #3702
• fix: test deadlock with processing the stdout of sed by @romange in #3735
• chore: add integrity checks to consumer->pel by @romange in #3754
• fix: make snapshotting process more responsive by @romange in #3759
• fix: tune test_replicaof_reject_on_load parameters by @romange in #3762
• chore: logs on assert fail for test_acl_cat_commands_multi_exec_squash by @kostasrim in #3749
• chore: add more error logs around ziplist parsing checks by @romange in #3764
• fix: substr/getrange result for invalid range by @BorysTheDev in #3766
• feat(zset_family): add ZRANGESTORE by @andydunstall in #3757
• fix server: fix last error reply by @adiholden in #3728
• chore: give up on InlinedVector due to spurious warnings with optional by @romange in #3765
• fix: mark pubusb commands as unsupported for cluster by @BorysTheDev in #3767
• chore: some renames + fix a typo in RETURN_ON_BAD_STATUS by @romange in #3763
• fix: setrange with empty value doesn't modify the DB by @BorysTheDev in #3771

New Contributors

• @howjmay made their first contribution in #3655
• @katapultcloud made their first contribution in https://githu...

v1.22.2

Dragonfly v1.22.2

This is a patch release.

fix(docker/healthcheck): run netstat port retreival command as dfly (#3647)
fix: data inconsistency on replica during replication (#3710)

v1.22.1

Dragonfly v1.22.1

This is a patch release. Urgency to update: low

fix: infinite recursive calls in the allocation tracker #3665

Full Changelog: v1.22.0...v1.22.1

v1.22.0

Dragonfly v1.22.0

Some prominent changes include:

• Lots of stability fixes around JSON and Search APIS:
  #3459, #3284, #3537, #3544,#3580, #3582, #3578, #3591, #3597, #3539
• Support HELLO AUTH handshake: #3486
• Implement FT.TAGVALS: #3493
• Introduce Dragonfly specific lua pragmas: #3517
• Memcache now replicates flags correctly: #3538
• Memcache API now supports absolute timestamps as well: #3607
• PEXPIRETIME and EXPIRETIME are implemented by @Eunoia1729 - thank you!
• Last but not least, a significant contribution by @cunla: a massive test suite, likely the largest in Dragonfly's history, with 12K lines of code. This contribution has sparked further efforts to enhance Dragonfly's compatibility and stability. The initial results are included in this release.

What's Changed

• fix: missing logs on ci timeout by @kostasrim in #3452
• fix: JSON.MSET command by @BorysTheDev in #3459
• chore: integrate quicklist changes from valkey by @romange in #3440
• feat(json_family): Add support of the JSON legacy mode by @BagritsevichStepan in #3284
• fix: pytest teardown exception in os.remove(LAST_LOGS) by @kostasrim in #3463
• chore: change how we track memory_budget during evictions by @romange in #3457
• chore: add timeout to replication sockets by @romange in #3434
• chore(server): Remove old blocking debug by @dranikpg in #3460
• chore: SinkReplyBuilder2 with vec batching by @dranikpg in #3454
• feat: Allow pre-declaring Lua SHAs to run with undeclared keys by @chakaz in #3465
• chore(server): Use parser more in list_family by @dranikpg in #3461
• fix: Make replica's REPLCONF IP-ADDRESS optional by @chakaz in #3473
• chore: make tiered_test more reliable by @romange in #3477
• chore(facade): Update SinkReplyBuilder2 by @dranikpg in #3474
• refactor: set info_replication_valkey_compatible=true by @BorysTheDev in #3467
• chore: relax the version check in test_replicate_old_master by @romange in #3478
• chore: Download Dragonfly versions only when needed by @chakaz in #3479
• fix: division by zero bug by @romange in #3482
• test: fix test_scripts by @BorysTheDev in #3485
• chore(facade): RedisReplyBuilder2 base by @dranikpg in #3475
• chore(facade): MCReplyBuilder2 by @dranikpg in #3480
• chore(facade): RedisReplyBuilder2 (extensions) by @dranikpg in #3481
• fix: big value serialization corner cases by @kostasrim in #3430
• fix: HELLO AUTH with non default user by @kostasrim in #3486
• fix: flake in teardown of test_unknown_dfly_env by @kostasrim in #3487
• fix: test_replicate_old_master missing serialization_max_chunk_size by @kostasrim in #3498
• chore: add logs to debug tiered memory failures by @romange in #3499
• feat: implement FT.TAGVALS by @romange in #3493
• test: fix test_acl_deluser by @BorysTheDev in #3488
• fix: sanitizers post failure on google chat by @kostasrim in #3496
• chore(generic_family): Fix bad data format error in the RESTORE command by @BagritsevichStepan in #3501
• fix: sanitizers false positives by @kostasrim in #3495
• feat: Expose replica_reconnect_count for Prometheus metrics by @BagritsevichStepan in #3370
• fix: pytest warnings by @kostasrim in #3497
• fix: missing virtual destructors in ReplyBuilder2 by @kostasrim in #3506
• chore: split engine_shard file from engine_shard_set by @romange in #3507
• chore: introduce a secondary TaskQueue for shards by @romange in #3508
• fix: clang warnings by @romange in #3509
• chore: reduce pipelining latency by reusing existing shard fibers by @romange in #3494
• fix: bugs around the growth of a tiered file by @romange in #3516
• chore: Change Lua embedded flags syntax by @chakaz in #3517
• feat(cluster): Allow appending RDB to existing store by @chakaz in #3505
• fix: disable ThreadLocalMutex when big value ser is off by @kostasrim in #3521
• chore(io): Introduce (carefully) new io with use_new_io flag by @dranikpg in #3513
• chore(io): Optimize repeated ReservePiece calls by @dranikpg in #3525
• fix: disable sanitizers false positive by @kostasrim in #3522
• chore(config): make pipeline_squash configurable by @dranikpg in #3529
• chore(traffic loger): use pipelining and print/analyze commands by @dranikpg in #3527
• fix: skip empty objects on load and replication by @kostasrim in #3514
• fix: deadlock in Heartbeat() by @kostasrim in #3530
• fix(search_family): Add error on creating indexes from non-zero databases by @BagritsevichStepan in #3537
• fix: macos build by @kostasrim in #3536
• fix: MC flags size and serialization #3134 by @BorysTheDev in #3538
• fix(transaction): Don't set continuation for blocking by @dranikpg in #3540
• fix: named volume permissions in docker by @romange in #3518
• fix(jsonpath): Add JsonPath grammar for the child identifier in brackets by @BagritsevichStepan in #3533
• fix: build errors in sanitizers daily workflow by @kostasrim in #3542
• test: tests using fakeredis to check compatibility by @cunla in #3534
• chore: fix search replication by @dranikpg in #3547
• chore: add deallocation logs in the allocation tracker by @romange in #3549
• fix: fakeredis tests support python 3.10 by @cunla in #3550
• feat(server_family): Add backup/restore Prometheus metrics by @BagritsevichStepan in #3520
• docs: Fakeredis tests README by @cunla in #3552
• fix: zinterstore correctly finds weights by @romange in #3554
• fix(search_family): Fix query parsing for the integer tags in OR expression by @BagritsevichStepan in #3544
• chore(server): Fix watch by @dranikpg in #3557
• chore(search): fix numeric index query in rev order by @dranikpg in #3555
• chore: allow limiting pipelining queue by length by @romange in #3551
• fix: add unsupported_server_types to fakeredis tests by @cunla in #3560
• fix: string compatibility issues by @romange in #3564
• fix(server): Fix ZRANGEBYLEX limit params by @dranikpg in #3562
• fix: compatibility around list and string commands by @romange in #3565
• chore(server): Unify zset arg parsing by @dranikpg in #3563
• fix: compatibility around list,string and sort commands by @romange in #3568
• chore(server): Sort correctly in ZINTER by @dranikpg in #3566
• chore(server): Unify zset boolean operations into single function by @dranikpg in #3567
• chore(transaction): More blocking tests by @dranikpg in #3546
• fix: delete empty dense sets in HGetGeneric by @kostasrim in #3543
• chore(search): Silence query parser error by @dranikpg in https:...

v1.21.4

Dragonfly v1.21.4

This is a patch release.

chore: fix search replication (#3547)

v1.21.3

Dragonfly v1.21.3

This is a patch release.

fix: display real replica remote address in INFO logs
fix: crash with concurrent multi-shard BLMOVE operations #3540

This follows up on Dragonfly v1.21.2 release.

v1.21.2

Dragonfly v1.21.2

This is a patch release.

fix: disable code that can cause potential deadlocks during the replication (#3521)

This follows up on Dragonfly v1.21.0 release, that includes the following prominent features:

• Alpha release of SSD Data tiering - enabled with flag --tiered_prefix some/path/basename
• Very basic support of multi-tenancy #3260
• HSETEX now supports NX option, see our docs.
• Added support for JSON.MERGE and for JSON.MSET.
• valkey's replica-announce-ip and --cluster--announc--ip are consolidated via --announce-ip flag.

Full Changelog: v1.21.1...v1.21.2