feat(tiering): Simple snapshotting #3073
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
dranikpg
commented
May 23, 2024
•
dranikpg
commented
- Adds support for snapshotting for tiered entries
dranikpg
force-pushed
the
tiering-snapshot
branch
from
187e094
to
2310a56
Compare
dranikpg
force-pushed
the
tiering-snapshot
branch
from
dd12e26
to
608bbd1
Compare
|
Ah dang, it conflicts with defragmentation 🤨 |
dranikpg
force-pushed
the
tiering-snapshot
branch
from
40a4f04
to
d52eefb
Compare
|
Now we don't fetch and don't offload while snapshotting is in progress. I alslo fixed the bug that occured when we did this |
romange
reviewed
Jun 4, 2024
src/server/snapshot.cc
Outdated
| } | ||
|
|
||
| bool SliceSnapshot::PushSerializedToChannel(bool force) { | ||
| // Bucket serialization might have accumulated some delayed values. | ||
| // Becuause we can finally block in this function, we'll await and serialize them |
src/server/snapshot.h
Outdated
| @@ -141,6 +150,7 @@ class SliceSnapshot { | |||
| DbIndex current_db_; | |||
|
|
|||
| std::unique_ptr<RdbSerializer> serializer_; | |||
| std::deque<DelayedEntry> delayed_entries_; // collected during atomic bucket traversal | |||
There was a problem hiding this comment.
why deque? vector (stack) won't work here?
dranikpg
force-pushed
the
tiering-snapshot
branch
from
b88d0e8
to
e5bd35a
Compare
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
romange
approved these changes
Jun 4, 2024
szinn
referenced
this pull request
in szinn/k8s-homelab
Jun 5, 2024
…nfly ( v1.18.1 → v1.19.0 ) (#3784) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [docker.dragonflydb.io/dragonflydb/dragonfly](https://togithub.com/dragonflydb/dragonfly) | minor | `v1.18.1` -> `v1.19.0` | --- ### Release Notes <details> <summary>dragonflydb/dragonfly (docker.dragonflydb.io/dragonflydb/dragonfly)</summary> ### [`v1.19.0`](https://togithub.com/dragonflydb/dragonfly/releases/tag/v1.19.0) [Compare Source](https://togithub.com/dragonflydb/dragonfly/compare/v1.18.1...v1.19.0) ##### Dragonfly v1.19.0 Some prominent changes include: - SPOP and SRANDMEMBER return truly randomized results ([#​3022](https://togithub.com/dragonflydb/dragonfly/issues/3022)) - Fix memory blowup in some cases during replication ([#​3084](https://togithub.com/dragonflydb/dragonfly/issues/3084) [#​3103](https://togithub.com/dragonflydb/dragonfly/issues/3103)) - Snapshotting works with data tiering ([#​3073](https://togithub.com/dragonflydb/dragonfly/issues/3073)) - Fix incompatibility issue of RDB snapshot with Redis 6.x ([#​3121](https://togithub.com/dragonflydb/dragonfly/issues/3121)) - Memory utilization improvements + better introspection of memory usage via /metrics ##### What's Changed - chore: update versions and remove caching code by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3030](https://togithub.com/dragonflydb/dragonfly/pull/3030) - chore: fix macos tiering by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3032](https://togithub.com/dragonflydb/dragonfly/pull/3032) - feat: support subrange expressions in jsonpathv2 by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3036](https://togithub.com/dragonflydb/dragonfly/pull/3036) - fix: reduce delay when stop replica [#​3020](https://togithub.com/dragonflydb/dragonfly/issues/3020) by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3028](https://togithub.com/dragonflydb/dragonfly/pull/3028) - fix(set): fix random in SRANDMEMBER and SPOP commands by [@​BagritsevichStepan](https://togithub.com/BagritsevichStepan) in [https://github.com/dragonflydb/dragonfly/pull/3022](https://togithub.com/dragonflydb/dragonfly/pull/3022) - chore: prevent updating gcc on macos by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3037](https://togithub.com/dragonflydb/dragonfly/pull/3037) - chore: add half-range indices by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3041](https://togithub.com/dragonflydb/dragonfly/pull/3041) - chore: update helio together with new mimalloc version by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3040](https://togithub.com/dragonflydb/dragonfly/pull/3040) - fix: header for oom_errors_total by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3043](https://togithub.com/dragonflydb/dragonfly/pull/3043) - chore: reset enable_direct_fd to avoid socket leakage in kernel by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3051](https://togithub.com/dragonflydb/dragonfly/pull/3051) - chore: update deprecated actions by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3052](https://togithub.com/dragonflydb/dragonfly/pull/3052) - chore: remove unneeded check-fails by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3054](https://togithub.com/dragonflydb/dragonfly/pull/3054) - chore: Export replication memory stats by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3062](https://togithub.com/dragonflydb/dragonfly/pull/3062) - <chore>!: Update grafana panel by [@​MaoMaoCake](https://togithub.com/MaoMaoCake) in [https://github.com/dragonflydb/dragonfly/pull/3064](https://togithub.com/dragonflydb/dragonfly/pull/3064) - chore: add replication memory stats to the dashboard by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3065](https://togithub.com/dragonflydb/dragonfly/pull/3065) - chore: bpop prints by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3076](https://togithub.com/dragonflydb/dragonfly/pull/3076) - fix(tiering): wait for IO before test teardown by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3079](https://togithub.com/dragonflydb/dragonfly/pull/3079) - chore: small replayer fixes by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3081](https://togithub.com/dragonflydb/dragonfly/pull/3081) - fix: increase lua stack limit to 8KB by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3080](https://togithub.com/dragonflydb/dragonfly/pull/3080) - chore: minor fixes by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3082](https://togithub.com/dragonflydb/dragonfly/pull/3082) - fix: fix cluster incorrect keys status by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3083](https://togithub.com/dragonflydb/dragonfly/pull/3083) - fix(acl): return -NOPERM instead of response error by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3049](https://togithub.com/dragonflydb/dragonfly/pull/3049) - feat(server): Expose serialization bytes via `INFO` and `/metrics` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3087](https://togithub.com/dragonflydb/dragonfly/pull/3087) - chore(streams): Some refactoring by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3088](https://togithub.com/dragonflydb/dragonfly/pull/3088) - feat(server): remove multi shard sync from replication by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3085](https://togithub.com/dragonflydb/dragonfly/pull/3085) - fix(replication): fullsync phase write to sync on noop by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3084](https://togithub.com/dragonflydb/dragonfly/pull/3084) - fix(tiering): Async delete for small bins by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3068](https://togithub.com/dragonflydb/dragonfly/pull/3068) - fix: prevent crashing if error happened during snapshot moving by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3092](https://togithub.com/dragonflydb/dragonfly/pull/3092) - fix: add timeout for DFLYMIGRATE ACK to prevent deadlock by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3093](https://togithub.com/dragonflydb/dragonfly/pull/3093) - chore: improve Migration() by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3033](https://togithub.com/dragonflydb/dragonfly/pull/3033) - feat(streams): Stream optimizations by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3091](https://togithub.com/dragonflydb/dragonfly/pull/3091) - feat(tiering): Defragmentation by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3021](https://togithub.com/dragonflydb/dragonfly/pull/3021) - fix: bit shift of kTagMask by [@​haodongnj](https://togithub.com/haodongnj) in [https://github.com/dragonflydb/dragonfly/pull/3099](https://togithub.com/dragonflydb/dragonfly/pull/3099) - feat(cluster_mgr): Allow attaching replicas by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3105](https://togithub.com/dragonflydb/dragonfly/pull/3105) - bug(server): fix replication stuck in full sync by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3103](https://togithub.com/dragonflydb/dragonfly/pull/3103) - fix(server): Sync FLUSH with tiering by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3098](https://togithub.com/dragonflydb/dragonfly/pull/3098) - chore: export pipeline related metrics by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3104](https://togithub.com/dragonflydb/dragonfly/pull/3104) - chore: zset error log (from 1.18 branch) by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3109](https://togithub.com/dragonflydb/dragonfly/pull/3109) - New test for cluster migration: connection issue by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3102](https://togithub.com/dragonflydb/dragonfly/pull/3102) - chore: minor fixes + remove redundant DCHECK by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3115](https://togithub.com/dragonflydb/dragonfly/pull/3115) - fix: fix cluster_fuzzy_migration test by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3116](https://togithub.com/dragonflydb/dragonfly/pull/3116) - feat(server): Pipeline and dispatch on `/metrics` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3117](https://togithub.com/dragonflydb/dragonfly/pull/3117) - chore: pull helio add test for tls deadlock by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3111](https://togithub.com/dragonflydb/dragonfly/pull/3111) - feat(cluster_mgr): Improvements to `cluster_mgr.py` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3118](https://togithub.com/dragonflydb/dragonfly/pull/3118) - refactor client tracking, fix atomicity, squashing and multi/exec by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/2970](https://togithub.com/dragonflydb/dragonfly/pull/2970) - fix(server): fix compatibility with rdb snapshot by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3121](https://togithub.com/dragonflydb/dragonfly/pull/3121) - Revert "refactor client tracking, fix atomicity, squashing and multi/… by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3122](https://togithub.com/dragonflydb/dragonfly/pull/3122) - feat(cluster_mgr): Take over command by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3120](https://togithub.com/dragonflydb/dragonfly/pull/3120) - fix: fix RestoreStreamer to prevent buckets skipping [#​2830](https://togithub.com/dragonflydb/dragonfly/issues/2830) by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3119](https://togithub.com/dragonflydb/dragonfly/pull/3119) - feat(cluster_mgr): Fix migration action by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3124](https://togithub.com/dragonflydb/dragonfly/pull/3124) - chore: prevent Dispatch fiber to be launched during migration by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3123](https://togithub.com/dragonflydb/dragonfly/pull/3123) - feat(tiering): Simple snapshotting by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3073](https://togithub.com/dragonflydb/dragonfly/pull/3073) - fix: upload action in version release by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3127](https://togithub.com/dragonflydb/dragonfly/pull/3127) ##### New Contributors - [@​MaoMaoCake](https://togithub.com/MaoMaoCake) made their first contribution in [https://github.com/dragonflydb/dragonfly/pull/3064](https://togithub.com/dragonflydb/dragonfly/pull/3064) - [@​haodongnj](https://togithub.com/haodongnj) made their first contribution in [https://github.com/dragonflydb/dragonfly/pull/3099](https://togithub.com/dragonflydb/dragonfly/pull/3099) ##### Huge thanks to all the contributors! ❤️ **Full Changelog**: dragonflydb/dragonfly@v1.18.0...v1.19.0 </details> <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zOTEuMCIsInVwZGF0ZWRJblZlciI6IjM3LjM5MS4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL21pbm9yIl19--> Co-authored-by: repo-jeeves[bot] <106431701+repo-jeeves[bot]@users.noreply.github.com>
kireque
referenced
this pull request
in kireque/home-ops
Jun 6, 2024
…nfly ( v1.18.1 → v1.19.0 ) (#629) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [docker.dragonflydb.io/dragonflydb/dragonfly](https://togithub.com/dragonflydb/dragonfly) | minor | `v1.18.1` -> `v1.19.0` | --- ### Release Notes <details> <summary>dragonflydb/dragonfly (docker.dragonflydb.io/dragonflydb/dragonfly)</summary> ### [`v1.19.0`](https://togithub.com/dragonflydb/dragonfly/releases/tag/v1.19.0) [Compare Source](https://togithub.com/dragonflydb/dragonfly/compare/v1.18.1...v1.19.0) ##### Dragonfly v1.19.0 Some prominent changes include: - SPOP and SRANDMEMBER return truly randomized results ([#​3022](https://togithub.com/dragonflydb/dragonfly/issues/3022)) - Fix memory blowup in some cases during replication ([#​3084](https://togithub.com/dragonflydb/dragonfly/issues/3084) [#​3103](https://togithub.com/dragonflydb/dragonfly/issues/3103)) - Snapshotting works with data tiering ([#​3073](https://togithub.com/dragonflydb/dragonfly/issues/3073)) - Fix incompatibility issue of RDB snapshot with Redis 6.x ([#​3121](https://togithub.com/dragonflydb/dragonfly/issues/3121)) - Memory utilization improvements + better introspection of memory usage via /metrics ##### What's Changed - chore: update versions and remove caching code by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3030](https://togithub.com/dragonflydb/dragonfly/pull/3030) - chore: fix macos tiering by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3032](https://togithub.com/dragonflydb/dragonfly/pull/3032) - feat: support subrange expressions in jsonpathv2 by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3036](https://togithub.com/dragonflydb/dragonfly/pull/3036) - fix: reduce delay when stop replica [#​3020](https://togithub.com/dragonflydb/dragonfly/issues/3020) by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3028](https://togithub.com/dragonflydb/dragonfly/pull/3028) - fix(set): fix random in SRANDMEMBER and SPOP commands by [@​BagritsevichStepan](https://togithub.com/BagritsevichStepan) in [https://github.com/dragonflydb/dragonfly/pull/3022](https://togithub.com/dragonflydb/dragonfly/pull/3022) - chore: prevent updating gcc on macos by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3037](https://togithub.com/dragonflydb/dragonfly/pull/3037) - chore: add half-range indices by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3041](https://togithub.com/dragonflydb/dragonfly/pull/3041) - chore: update helio together with new mimalloc version by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3040](https://togithub.com/dragonflydb/dragonfly/pull/3040) - fix: header for oom_errors_total by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3043](https://togithub.com/dragonflydb/dragonfly/pull/3043) - chore: reset enable_direct_fd to avoid socket leakage in kernel by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3051](https://togithub.com/dragonflydb/dragonfly/pull/3051) - chore: update deprecated actions by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3052](https://togithub.com/dragonflydb/dragonfly/pull/3052) - chore: remove unneeded check-fails by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3054](https://togithub.com/dragonflydb/dragonfly/pull/3054) - chore: Export replication memory stats by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3062](https://togithub.com/dragonflydb/dragonfly/pull/3062) - <chore>!: Update grafana panel by [@​MaoMaoCake](https://togithub.com/MaoMaoCake) in [https://github.com/dragonflydb/dragonfly/pull/3064](https://togithub.com/dragonflydb/dragonfly/pull/3064) - chore: add replication memory stats to the dashboard by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3065](https://togithub.com/dragonflydb/dragonfly/pull/3065) - chore: bpop prints by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3076](https://togithub.com/dragonflydb/dragonfly/pull/3076) - fix(tiering): wait for IO before test teardown by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3079](https://togithub.com/dragonflydb/dragonfly/pull/3079) - chore: small replayer fixes by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3081](https://togithub.com/dragonflydb/dragonfly/pull/3081) - fix: increase lua stack limit to 8KB by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3080](https://togithub.com/dragonflydb/dragonfly/pull/3080) - chore: minor fixes by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3082](https://togithub.com/dragonflydb/dragonfly/pull/3082) - fix: fix cluster incorrect keys status by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3083](https://togithub.com/dragonflydb/dragonfly/pull/3083) - fix(acl): return -NOPERM instead of response error by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3049](https://togithub.com/dragonflydb/dragonfly/pull/3049) - feat(server): Expose serialization bytes via `INFO` and `/metrics` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3087](https://togithub.com/dragonflydb/dragonfly/pull/3087) - chore(streams): Some refactoring by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3088](https://togithub.com/dragonflydb/dragonfly/pull/3088) - feat(server): remove multi shard sync from replication by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3085](https://togithub.com/dragonflydb/dragonfly/pull/3085) - fix(replication): fullsync phase write to sync on noop by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3084](https://togithub.com/dragonflydb/dragonfly/pull/3084) - fix(tiering): Async delete for small bins by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3068](https://togithub.com/dragonflydb/dragonfly/pull/3068) - fix: prevent crashing if error happened during snapshot moving by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3092](https://togithub.com/dragonflydb/dragonfly/pull/3092) - fix: add timeout for DFLYMIGRATE ACK to prevent deadlock by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3093](https://togithub.com/dragonflydb/dragonfly/pull/3093) - chore: improve Migration() by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3033](https://togithub.com/dragonflydb/dragonfly/pull/3033) - feat(streams): Stream optimizations by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3091](https://togithub.com/dragonflydb/dragonfly/pull/3091) - feat(tiering): Defragmentation by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3021](https://togithub.com/dragonflydb/dragonfly/pull/3021) - fix: bit shift of kTagMask by [@​haodongnj](https://togithub.com/haodongnj) in [https://github.com/dragonflydb/dragonfly/pull/3099](https://togithub.com/dragonflydb/dragonfly/pull/3099) - feat(cluster_mgr): Allow attaching replicas by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3105](https://togithub.com/dragonflydb/dragonfly/pull/3105) - bug(server): fix replication stuck in full sync by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3103](https://togithub.com/dragonflydb/dragonfly/pull/3103) - fix(server): Sync FLUSH with tiering by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3098](https://togithub.com/dragonflydb/dragonfly/pull/3098) - chore: export pipeline related metrics by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3104](https://togithub.com/dragonflydb/dragonfly/pull/3104) - chore: zset error log (from 1.18 branch) by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3109](https://togithub.com/dragonflydb/dragonfly/pull/3109) - New test for cluster migration: connection issue by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3102](https://togithub.com/dragonflydb/dragonfly/pull/3102) - chore: minor fixes + remove redundant DCHECK by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3115](https://togithub.com/dragonflydb/dragonfly/pull/3115) - fix: fix cluster_fuzzy_migration test by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3116](https://togithub.com/dragonflydb/dragonfly/pull/3116) - feat(server): Pipeline and dispatch on `/metrics` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3117](https://togithub.com/dragonflydb/dragonfly/pull/3117) - chore: pull helio add test for tls deadlock by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3111](https://togithub.com/dragonflydb/dragonfly/pull/3111) - feat(cluster_mgr): Improvements to `cluster_mgr.py` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3118](https://togithub.com/dragonflydb/dragonfly/pull/3118) - refactor client tracking, fix atomicity, squashing and multi/exec by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/2970](https://togithub.com/dragonflydb/dragonfly/pull/2970) - fix(server): fix compatibility with rdb snapshot by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3121](https://togithub.com/dragonflydb/dragonfly/pull/3121) - Revert "refactor client tracking, fix atomicity, squashing and multi/… by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3122](https://togithub.com/dragonflydb/dragonfly/pull/3122) - feat(cluster_mgr): Take over command by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3120](https://togithub.com/dragonflydb/dragonfly/pull/3120) - fix: fix RestoreStreamer to prevent buckets skipping [#​2830](https://togithub.com/dragonflydb/dragonfly/issues/2830) by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3119](https://togithub.com/dragonflydb/dragonfly/pull/3119) - feat(cluster_mgr): Fix migration action by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3124](https://togithub.com/dragonflydb/dragonfly/pull/3124) - chore: prevent Dispatch fiber to be launched during migration by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3123](https://togithub.com/dragonflydb/dragonfly/pull/3123) - feat(tiering): Simple snapshotting by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3073](https://togithub.com/dragonflydb/dragonfly/pull/3073) - fix: upload action in version release by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3127](https://togithub.com/dragonflydb/dragonfly/pull/3127) ##### New Contributors - [@​MaoMaoCake](https://togithub.com/MaoMaoCake) made their first contribution in [https://github.com/dragonflydb/dragonfly/pull/3064](https://togithub.com/dragonflydb/dragonfly/pull/3064) - [@​haodongnj](https://togithub.com/haodongnj) made their first contribution in [https://github.com/dragonflydb/dragonfly/pull/3099](https://togithub.com/dragonflydb/dragonfly/pull/3099) ##### Huge thanks to all the contributors! ❤️ **Full Changelog**: dragonflydb/dragonfly@v1.18.0...v1.19.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zOTAuMSIsInVwZGF0ZWRJblZlciI6IjM3LjM5MC4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL21pbm9yIl19--> Co-authored-by: kireque-bot[bot] <143391978+kireque-bot[bot]@users.noreply.github.com>
lumiere-bot bot
referenced
this pull request
in coolguy1771/home-ops
Jun 7, 2024
…19.0 ) (#4821) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/dragonflydb/dragonfly](https://togithub.com/dragonflydb/dragonfly) | minor | `v1.18.1` -> `v1.19.0` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>dragonflydb/dragonfly (ghcr.io/dragonflydb/dragonfly)</summary> ### [`v1.19.0`](https://togithub.com/dragonflydb/dragonfly/releases/tag/v1.19.0) [Compare Source](https://togithub.com/dragonflydb/dragonfly/compare/v1.18.1...v1.19.0) ##### Dragonfly v1.19.0 Some prominent changes include: - SPOP and SRANDMEMBER return truly randomized results ([#​3022](https://togithub.com/dragonflydb/dragonfly/issues/3022)) - Fix memory blowup in some cases during replication ([#​3084](https://togithub.com/dragonflydb/dragonfly/issues/3084) [#​3103](https://togithub.com/dragonflydb/dragonfly/issues/3103)) - Snapshotting works with data tiering ([#​3073](https://togithub.com/dragonflydb/dragonfly/issues/3073)) - Fix incompatibility issue of RDB snapshot with Redis 6.x ([#​3121](https://togithub.com/dragonflydb/dragonfly/issues/3121)) - Memory utilization improvements + better introspection of memory usage via /metrics ##### What's Changed - chore: update versions and remove caching code by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3030](https://togithub.com/dragonflydb/dragonfly/pull/3030) - chore: fix macos tiering by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3032](https://togithub.com/dragonflydb/dragonfly/pull/3032) - feat: support subrange expressions in jsonpathv2 by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3036](https://togithub.com/dragonflydb/dragonfly/pull/3036) - fix: reduce delay when stop replica [#​3020](https://togithub.com/dragonflydb/dragonfly/issues/3020) by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3028](https://togithub.com/dragonflydb/dragonfly/pull/3028) - fix(set): fix random in SRANDMEMBER and SPOP commands by [@​BagritsevichStepan](https://togithub.com/BagritsevichStepan) in [https://github.com/dragonflydb/dragonfly/pull/3022](https://togithub.com/dragonflydb/dragonfly/pull/3022) - chore: prevent updating gcc on macos by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3037](https://togithub.com/dragonflydb/dragonfly/pull/3037) - chore: add half-range indices by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3041](https://togithub.com/dragonflydb/dragonfly/pull/3041) - chore: update helio together with new mimalloc version by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3040](https://togithub.com/dragonflydb/dragonfly/pull/3040) - fix: header for oom_errors_total by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3043](https://togithub.com/dragonflydb/dragonfly/pull/3043) - chore: reset enable_direct_fd to avoid socket leakage in kernel by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3051](https://togithub.com/dragonflydb/dragonfly/pull/3051) - chore: update deprecated actions by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3052](https://togithub.com/dragonflydb/dragonfly/pull/3052) - chore: remove unneeded check-fails by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3054](https://togithub.com/dragonflydb/dragonfly/pull/3054) - chore: Export replication memory stats by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3062](https://togithub.com/dragonflydb/dragonfly/pull/3062) - <chore>!: Update grafana panel by [@​MaoMaoCake](https://togithub.com/MaoMaoCake) in [https://github.com/dragonflydb/dragonfly/pull/3064](https://togithub.com/dragonflydb/dragonfly/pull/3064) - chore: add replication memory stats to the dashboard by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3065](https://togithub.com/dragonflydb/dragonfly/pull/3065) - chore: bpop prints by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3076](https://togithub.com/dragonflydb/dragonfly/pull/3076) - fix(tiering): wait for IO before test teardown by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3079](https://togithub.com/dragonflydb/dragonfly/pull/3079) - chore: small replayer fixes by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3081](https://togithub.com/dragonflydb/dragonfly/pull/3081) - fix: increase lua stack limit to 8KB by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3080](https://togithub.com/dragonflydb/dragonfly/pull/3080) - chore: minor fixes by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3082](https://togithub.com/dragonflydb/dragonfly/pull/3082) - fix: fix cluster incorrect keys status by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3083](https://togithub.com/dragonflydb/dragonfly/pull/3083) - fix(acl): return -NOPERM instead of response error by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3049](https://togithub.com/dragonflydb/dragonfly/pull/3049) - feat(server): Expose serialization bytes via `INFO` and `/metrics` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3087](https://togithub.com/dragonflydb/dragonfly/pull/3087) - chore(streams): Some refactoring by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3088](https://togithub.com/dragonflydb/dragonfly/pull/3088) - feat(server): remove multi shard sync from replication by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3085](https://togithub.com/dragonflydb/dragonfly/pull/3085) - fix(replication): fullsync phase write to sync on noop by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3084](https://togithub.com/dragonflydb/dragonfly/pull/3084) - fix(tiering): Async delete for small bins by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3068](https://togithub.com/dragonflydb/dragonfly/pull/3068) - fix: prevent crashing if error happened during snapshot moving by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3092](https://togithub.com/dragonflydb/dragonfly/pull/3092) - fix: add timeout for DFLYMIGRATE ACK to prevent deadlock by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3093](https://togithub.com/dragonflydb/dragonfly/pull/3093) - chore: improve Migration() by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3033](https://togithub.com/dragonflydb/dragonfly/pull/3033) - feat(streams): Stream optimizations by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3091](https://togithub.com/dragonflydb/dragonfly/pull/3091) - feat(tiering): Defragmentation by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3021](https://togithub.com/dragonflydb/dragonfly/pull/3021) - fix: bit shift of kTagMask by [@​haodongnj](https://togithub.com/haodongnj) in [https://github.com/dragonflydb/dragonfly/pull/3099](https://togithub.com/dragonflydb/dragonfly/pull/3099) - feat(cluster_mgr): Allow attaching replicas by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3105](https://togithub.com/dragonflydb/dragonfly/pull/3105) - bug(server): fix replication stuck in full sync by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3103](https://togithub.com/dragonflydb/dragonfly/pull/3103) - fix(server): Sync FLUSH with tiering by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3098](https://togithub.com/dragonflydb/dragonfly/pull/3098) - chore: export pipeline related metrics by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3104](https://togithub.com/dragonflydb/dragonfly/pull/3104) - chore: zset error log (from 1.18 branch) by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3109](https://togithub.com/dragonflydb/dragonfly/pull/3109) - New test for cluster migration: connection issue by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3102](https://togithub.com/dragonflydb/dragonfly/pull/3102) - chore: minor fixes + remove redundant DCHECK by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3115](https://togithub.com/dragonflydb/dragonfly/pull/3115) - fix: fix cluster_fuzzy_migration test by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3116](https://togithub.com/dragonflydb/dragonfly/pull/3116) - feat(server): Pipeline and dispatch on `/metrics` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3117](https://togithub.com/dragonflydb/dragonfly/pull/3117) - chore: pull helio add test for tls deadlock by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/3111](https://togithub.com/dragonflydb/dragonfly/pull/3111) - feat(cluster_mgr): Improvements to `cluster_mgr.py` by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3118](https://togithub.com/dragonflydb/dragonfly/pull/3118) - refactor client tracking, fix atomicity, squashing and multi/exec by [@​kostasrim](https://togithub.com/kostasrim) in [https://github.com/dragonflydb/dragonfly/pull/2970](https://togithub.com/dragonflydb/dragonfly/pull/2970) - fix(server): fix compatibility with rdb snapshot by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3121](https://togithub.com/dragonflydb/dragonfly/pull/3121) - Revert "refactor client tracking, fix atomicity, squashing and multi/… by [@​adiholden](https://togithub.com/adiholden) in [https://github.com/dragonflydb/dragonfly/pull/3122](https://togithub.com/dragonflydb/dragonfly/pull/3122) - feat(cluster_mgr): Take over command by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3120](https://togithub.com/dragonflydb/dragonfly/pull/3120) - fix: fix RestoreStreamer to prevent buckets skipping [#​2830](https://togithub.com/dragonflydb/dragonfly/issues/2830) by [@​BorysTheDev](https://togithub.com/BorysTheDev) in [https://github.com/dragonflydb/dragonfly/pull/3119](https://togithub.com/dragonflydb/dragonfly/pull/3119) - feat(cluster_mgr): Fix migration action by [@​chakaz](https://togithub.com/chakaz) in [https://github.com/dragonflydb/dragonfly/pull/3124](https://togithub.com/dragonflydb/dragonfly/pull/3124) - chore: prevent Dispatch fiber to be launched during migration by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3123](https://togithub.com/dragonflydb/dragonfly/pull/3123) - feat(tiering): Simple snapshotting by [@​dranikpg](https://togithub.com/dranikpg) in [https://github.com/dragonflydb/dragonfly/pull/3073](https://togithub.com/dragonflydb/dragonfly/pull/3073) - fix: upload action in version release by [@​romange](https://togithub.com/romange) in [https://github.com/dragonflydb/dragonfly/pull/3127](https://togithub.com/dragonflydb/dragonfly/pull/3127) ##### New Contributors - [@​MaoMaoCake](https://togithub.com/MaoMaoCake) made their first contribution in [https://github.com/dragonflydb/dragonfly/pull/3064](https://togithub.com/dragonflydb/dragonfly/pull/3064) - [@​haodongnj](https://togithub.com/haodongnj) made their first contribution in [https://github.com/dragonflydb/dragonfly/pull/3099](https://togithub.com/dragonflydb/dragonfly/pull/3099) ##### Huge thanks to all the contributors! ❤️ **Full Changelog**: dragonflydb/dragonfly@v1.18.0...v1.19.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zOTAuMSIsInVwZGF0ZWRJblZlciI6IjM3LjM5MC4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL21pbm9yIl19--> Co-authored-by: lumiere-bot[bot] <98047013+lumiere-bot[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.