feat(server): support cluster replication #2748

adiholden · 2024-03-19T15:43:15Z

In this PR we introduce new command
ADDREPLICAOF
and extend REPLICAOF command with optional params

The slot range data is used only to flush the relevant data if the replication to a node disconnected

To replicate a cluster we first issue command
REPLICAOF
with data for a single node from the cluster
than for each additinal cluster master node we run
ADDREPLICAOF

To promote the replica to master we will run
REPLICAOF NO ONE

If we run REPLICAOF this will start replication from scratch removing all added replicas

romange · 2024-03-19T16:12:25Z

@adiholden please rebase

chakaz

Very nice!

chakaz · 2024-03-20T04:45:41Z

src/server/cluster/slot_set.h

+    std::string slots_str;
+    for (SlotId i = 0; i < kSlotsNumber; ++i) {
+      if (slots_->test(i)) {
+        absl::StrAppend(&slots_str, absl::StrCat(i));


Don't we need a separator here? A comma or something?

Perhaps a space, as you use it to build commands

you are right, I will also need to add a pytest tests to make sure we actually flush the correct slots

chakaz · 2024-03-20T04:49:23Z

src/server/journal/executor.cc

+void JournalExecutor::FlushSlots(const SlotRange& slot_range) {
+  SlotRanges slot_ranges = {slot_range};
+  SlotSet slot_set(slot_ranges);
+  auto cmd = BuildFromParts("FLUSHSLOTS", slot_set.ToString());


This string is likely to be pretty huge. We could modify FLUSHSLOTS to accept slot ranges, but perhaps that's over engineering

I don't think sending two numbers instead of up to 16 thousand is over-engineering 😆

I agree with you that we need to move to slot ranges, but this is a big change.. I would like to do it in a different PR as this one is already big enough

chakaz · 2024-03-20T04:54:52Z

src/server/replica.cc

@@ -375,13 +376,17 @@ error_code Replica::InitiatePSync() {
    io::PrefixSource ps{io_buf.InputBuffer(), Sock()};

    // Set LOADING state.
-    CHECK(service_.SwitchState(GlobalState::ACTIVE, GlobalState::LOADING).first ==
-          GlobalState::LOADING);
+    CHECK(service_.SwitchState(GlobalState::ACTIVE, GlobalState::LOADING) == GlobalState::LOADING);


CHECK_EQ() for better logging? 🙏

chakaz · 2024-03-20T04:55:29Z

src/server/replica.cc

@@ -466,8 +471,7 @@ error_code Replica::InitiateDflySync() {
  RETURN_ON_ERR(cntx_.SwitchErrorHandler(std::move(err_handler)));

  // Make sure we're in LOADING state.
-  CHECK(service_.SwitchState(GlobalState::ACTIVE, GlobalState::LOADING).first ==
-        GlobalState::LOADING);
+  CHECK(service_.SwitchState(GlobalState::ACTIVE, GlobalState::LOADING) == GlobalState::LOADING);


chakaz · 2024-03-20T04:56:31Z

src/server/server_family.cc

@@ -554,6 +550,58 @@ string_view GetRedisMode() {
  return ClusterConfig::IsEnabledOrEmulated() ? "cluster"sv : "standalone"sv;
 }

+struct ReplicaOfArgs {
+  bool no_one = false;


I'd make this a method instead of a member. That could remove impossible states.

chakaz · 2024-03-20T05:01:04Z

src/server/server_family.cc

+  bool no_one = false;
+  string_view host;
+  string_view port_sv;
+  uint32_t port;


uint16_t?

chakaz · 2024-03-20T05:03:18Z

src/server/server_family.cc

+    return replicaof_args;
+  }
+
+  if (!absl::SimpleAtoi(replicaof_args.port_sv, &replicaof_args.port) || replicaof_args.port < 1 ||


If you use a uint16_t above you could change the range check to simply be replicaof_arg.port == 0

I dont see absl::SimpleAtoi impl with int16_t
do you have any utility function that we can use ?

You're right. I don't know why absl made that decision, but yeah, we need to use a temporary and check its upper range it looks like :|

chakaz · 2024-03-20T05:06:50Z

src/server/server_family.cc

+    if (!absl::SimpleAtoi(*slot_end, &slot_id_end)) {
+      return facade::OpStatus::INVALID_INT;
+    }
+    if (slot_id_end > ClusterConfig::kMaxSlotNum) {


Suggested change

if (slot_id_end > ClusterConfig::kMaxSlotNum) {

if ((slot_id_end > ClusterConfig::kMaxSlotNum) || (slot_id_end < slot_id_start)) {

we should really have a utility function for parsing slots or slot pairs in cluster code 😅

chakaz · 2024-03-20T05:12:07Z

src/server/server_family.cc

  if (replica_)
    replica_->Stop();
+  // Stop all cluster replication.
+  for (auto& replica : cluster_replicas_) {
+    replica->Stop();
+  }
+  cluster_replicas_.clear();


This can be wrapped in a small utility function
btw don't we need to call replica_.reset() like above?

chakaz · 2024-03-20T05:15:00Z

src/server/server_family.cc

-    Replica::Info rinfo = replica_->GetInfo();
-    rb->StartArray(4);
+
+    size_t additional_replication = cluster_replicas_.size();


I'd drop additional_replication and use the size() method inline, I don't think it adds clarity

dranikpg · 2024-03-20T12:26:40Z

src/server/journal/executor.cc

+void JournalExecutor::FlushSlots(const SlotRange& slot_range) {
+  SlotRanges slot_ranges = {slot_range};
+  SlotSet slot_set(slot_ranges);
+  auto cmd = BuildFromParts("FLUSHSLOTS", slot_set.ToString());


I don't think sending two numbers instead of up to 16 thousand is over-engineering 😆

dranikpg · 2024-03-20T12:35:04Z

src/server/main_service.cc

+  if (global_state_ == to && global_state_ == GlobalState::LOADING) {
+    ++loading_state_counter_;
+    return global_state_;
+  }
+  if (global_state_ == from && global_state_ == GlobalState::LOADING &&
+      loading_state_counter_ > 0) {
+    --loading_state_counter_;
+    if (loading_state_counter_ > 0) {
+      return global_state_;
+    }
+  }
+
+  if (global_state_ != from) {
+    return global_state_;
+  }


Nice trick to make it compatible. It only feels a little bit complicated because we have implicit rules which are stricter than the interface, i.e. for LOADING the from parameter has no meaning. So maybe it could have been handled by a function like RequestLoadingState() or smth.... Just a nit 🙂

for LOADING the from parameter is used as if we are not already in LOADING state we will change the state to LOADING only if global_state_ == from
so if we are in TAKEOVE/SHUTDOWN we will not change to loading because we ask to move from ACTIVE to LOADING

After looking more into the code, I think that we dont realy need the from param in the switch state as we will not be able to issue commands if we are not in active mode
I will do the change we RequestLoadingState and will later revisit the SwitchState function in a different PR

dranikpg · 2024-03-20T12:37:31Z

src/server/server_family.cc

+    if (!absl::SimpleAtoi(*slot_end, &slot_id_end)) {
+      return facade::OpStatus::INVALID_INT;
+    }
+    if (slot_id_end > ClusterConfig::kMaxSlotNum) {


we should really have a utility function for parsing slots or slot pairs in cluster code 😅

dranikpg · 2024-03-20T12:38:19Z

src/server/server_family.cc

+      for (auto& replica : cluster_replicas_) {
+        replica->Stop();
+      }


I assume this needs not to be embedded into if (replica_)?

I can move this outside but cluster_replicas_ will be not empty only if replica_ is set

dranikpg · 2024-03-20T12:58:27Z

src/server/server_family.h

  std::shared_ptr<Replica> replica_ ABSL_GUARDED_BY(replicaof_mu_);
+  std::vector<std::unique_ptr<Replica>> cluster_replicas_
+      ABSL_GUARDED_BY(replicaof_mu_);  // used to replicating multiple nodes to single dragonfly


It feels like replica_ is the leading node, whereas in reality they're equal. Either way it's not worth generalizing if it's just ad-hoc

Signed-off-by: adi_holden <adi@dragonflydb.io>

chakaz · 2024-03-25T07:13:03Z

src/server/cluster/slot_set.h

+    std::string slots_str;
+    for (SlotId i = 0; i < kSlotsNumber; ++i) {
+      if (slots_->test(i)) {
+        absl::StrAppend(&slots_str, absl::StrCat(i, " "));


Suggested change

absl::StrAppend(&slots_str, absl::StrCat(i, " "));

absl::StrAppend(&slots_str, i, " ");

chakaz · 2024-03-25T07:14:16Z

src/server/cluster/slot_set.h

+      if (slots_->test(i)) {
+        absl::StrAppend(&slots_str, absl::StrCat(i, " "));
+      }
+    }


Suggested change

}

}

if (!slots_str.empty()) {

slots_str.pop_back();

}

chakaz · 2024-03-25T07:16:35Z

src/server/cluster/slot_set.h

@@ -93,6 +95,16 @@ class SlotSet {
    return res;
  }

+  std::string ToString() const {


Do you use this? It looks like the new approach doesn't require this, no?

true will remove

chakaz · 2024-03-25T07:21:37Z

src/server/common.h

@@ -160,6 +160,12 @@ enum class GlobalState : uint8_t {
  TAKEN_OVER,
 };

+const char* GlobalStateName(GlobalState gs);


If you're modifying this, consider returning string_view

chakaz · 2024-03-25T07:22:50Z

src/server/debugcmd.cc

-    LOG(WARNING) << GlobalStateName(new_state.first) << " in progress, ignored";
+
+  if (new_state != GlobalState::LOADING) {
+    LOG(WARNING) << GlobalStateName(new_state) << " in progress, ignored";


Suggested change

LOG(WARNING) << GlobalStateName(new_state) << " in progress, ignored";

LOG(WARNING) << new_state << " in progress, ignored";

Isn't this why you added the above operator<<?

I added the operator because CHECK_EQ of 2 states did not work as it requires operator<<

chakaz · 2024-03-25T07:24:25Z

src/server/main_service.h

  GlobalState global_state_ = GlobalState::ACTIVE;  // protected by mu_;
+  uint32_t loading_state_counter_ = 0;              // protected by mu_;


Suggested change

GlobalState global_state_ = GlobalState::ACTIVE; // protected by mu_;

uint32_t loading_state_counter_ = 0; // protected by mu_;

GlobalState global_state_ ABSL_GUARDED_BY(my_) = GlobalState::ACTIVE;;

uint32_t loading_state_counter_ ABSL_GUARDED_BY(my_) = 0;

chakaz · 2024-03-25T07:31:59Z

src/server/server_family.cc

+struct ReplicaOfArgs {
+  bool no_one = false;
+  string_view host;
+  string_view port_sv;


I don't follow, sorry.
Why not have uint16_t and have it be 0 in case is it replica of no-one?
Then instead of port_sv member you could have a std::string GetPortString(), which will not allocate memory because of small string optimization (but in any case it's not important in this flow)

chakaz · 2024-03-25T07:32:14Z

src/server/server_family.cc

+struct ReplicaOfArgs {
+  bool no_one = false;
+  string_view host;
+  string_view port_sv;


Not that this is critical :) It's a nit.

chakaz · 2024-03-25T07:33:57Z

src/server/server_family.cc

-  if (new_state.first != GlobalState::LOADING) {
-    LOG(WARNING) << GlobalStateName(new_state.first) << " in progress, ignored";
+  if (new_state != GlobalState::LOADING) {
+    LOG(WARNING) << GlobalStateName(new_state) << " in progress, ignored";


Suggested change

LOG(WARNING) << GlobalStateName(new_state) << " in progress, ignored";

LOG(WARNING) << new_state << " in progress, ignored";

chakaz · 2024-03-25T07:39:40Z

src/server/server_family.cc

+    return cntx->SendError(replicaof_args.status());
+  }
+  if (replicaof_args->IsReplicaOfNoOne()) {
+    return cntx->SendError("ADDREPLICAOF does not supprot no one");


Suggested change

return cntx->SendError("ADDREPLICAOF does not supprot no one");

return cntx->SendError("ADDREPLICAOF does not support no one");

Signed-off-by: adi_holden <adi@dragonflydb.io>

chakaz · 2024-03-26T05:58:33Z

src/server/server_family.cc

+      return nullopt;
+    }
+    if (parser.HasNext()) {
+      auto [slot_start, slot_end] = parser.Next<SlotId, SlotId>();


Don't we need to check if parser has an error here?

It will return 0 for invalid integers, so we can have a spurious "invalid slot range" which is not a problem I guess 🤷🏻‍♂️

Why is [0, 0] an invalid range? We should be able to move the single slot 0.. and ranges are inclusive

it's not, but it also can be [x 0] 🙂

Signed-off-by: adi_holden <adi@dragonflydb.io>

…nfly ( v1.15.1 → v1.16.0 ) (#3354) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [docker.dragonflydb.io/dragonflydb/dragonfly](https://togithub.com/dragonflydb/dragonfly) | minor | `v1.15.1` -> `v1.16.0` | --- ### Release Notes <details> <summary>dragonflydb/dragonfly (docker.dragonflydb.io/dragonflydb/dragonfly)</summary> ### [`v1.16.0`](https://togithub.com/dragonflydb/dragonfly/releases/tag/v1.16.0) [Compare Source](https://togithub.com/dragonflydb/dragonfly/compare/v1.15.1...v1.16.0) ##### Dragonfly v1.16.0 Our spring release. We are getting closer to 2.0 with some very exciting features ahead. Stay tuned! Some prominent changes include: - Improved memory accounting of client connections ([#2710](https://togithub.com/dragonflydb/dragonfly/issues/2710) [#2755](https://togithub.com/dragonflydb/dragonfly/issues/2755) and [#2692](https://togithub.com/dragonflydb/dragonfly/issues/2692) ) - FT.AGGREGATE call ([#2413](https://togithub.com/dragonflydb/dragonfly/issues/2413)) - Properly handle and replicate Memcache flags ([#2787](https://togithub.com/dragonflydb/dragonfly/issues/2787) [#2807](https://togithub.com/dragonflydb/dragonfly/issues/2807)) - Intoduce BF.AGGREGATE BD.(M)ADD and BF.(M)EXISTS methods ([#2801](https://togithub.com/dragonflydb/dragonfly/issues/2801)). Note, that it does not work with snapshots and replication yet. - Dragonfly builds natively on MacOS. We would love some help with extending the release pipeline to create a proper macos binary. - Following the requests from the Edge developers community, we added a basic HTTP API support! Try running Dragonfly with: `--expose_http_api` flag and then call `curl -X POST -d '["ping"]' localhost:6379/api`. We will follow up with more extensive docs later this month. - Lots of stability fixes, especially around Sidekiq and BullMQ workloads. ##### What's Changed - chore: make usan asan optional and enable them on CI by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2631 - fix: missing manual trigger for daily sanitizers by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2682 - bug(tiering): fix overflow in page offset calculation and wrong hash offset calculation by [@theyueli](https://togithub.com/theyueli) in [dragonflydb/dragonfly#2683 - Chore: Fixed Docker Health Check by [@manojks1999](https://togithub.com/manojks1999) in [dragonflydb/dragonfly#2659 - chore: Increase disk space in the coverage runs by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2684 - fix(flushall): Decommit memory after releasing tables. by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2691 - feat(server): Account for serializer's temporary buffer size by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2689 - refactor: remove FULL-SYNC-CUT cmd [#2687](https://togithub.com/dragonflydb/dragonfly/issues/2687) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2688 - chore: add malloc-based stats and decommit by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2692 - feat(cluster): automatic slot migration finalization [#2697](https://togithub.com/dragonflydb/dragonfly/issues/2697) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2698 - Basic FT.AGGREGATE by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2413 - chore: little transaction cleanup by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2608 - fix(channel store): add acquire/release pair in fast update path by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2704 - chore: add ubuntu22 devcontainer by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2700 - feat(cluster): Add `--cluster_id` flag by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2695 - feat(server): Use mimalloc in SSL calls by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2710 - chore: Pull helio with new BlockingCounter by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2711 - chore(transaction): Simplify PollExecution by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2712 - chore(transaction): Don't call GetLocalMask from blocking controller by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2715 - chore: improve compatibility of EXPIRE functions with Redis by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2696 - chore: disable flaky fuzzy migration test by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2716 - chore: update sanitizers workflow by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2686 - chore: Use c-ares for resolving hosts in `ProtocolClient` by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2719 - Remove check-fail in ExpireIfNeeded and introduce DFLY LOAD by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2699 - chore: Record cmd stat from invoke by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2720 - fix(transaction): nullptr access on non transactional commands by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2724 - fix(BgSave): async from sync by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2702 - chore: remove core/fibers by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2723 - fix(transaction): Replace with armed sync point by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2708 - feat(json): Deserialize ReJSON format by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2725 - feat: add flag masteruser by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2693 - refactor: block list refactoring [#2580](https://togithub.com/dragonflydb/dragonfly/issues/2580) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2732 - chore: fix DeduceExecMode by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2733 - fix(cluster): Reply with correct `\n` / `\r\n` from `CLUSTER` sub cmd by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2731 - chore: Introduce fiber stack allocator by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2730 - fix(cluster): Save replica ID per replica by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2735 - fix(ssl): Proper cleanup by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2742 - chore: add skeleton files for flat_dfs code by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2738 - chore: better error reporting when connecting to tls with plain socket by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2740 - chore: Support json paths without root selector by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2747 - chore: journal cleanup by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2749 - refactor: remove start-slot-migration cmd [#2727](https://togithub.com/dragonflydb/dragonfly/issues/2727) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2728 - feat(server): Add TLS usage to /metrics and `INFO MEMORY` by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2755 - chore: preparations for adding flat json support by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2752 - chore(transaction): Introduce RunCallback by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2760 - feat(replication): Do not auto replicate different master by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2753 - Improve Helm chart to be rendered locally and on machines where is not the application target by [@fafg](https://togithub.com/fafg) in [dragonflydb/dragonfly#2706 - chore: preparation for basic http api by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2764 - feat(server): Add metric for RDB save duration. by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2768 - chore: fix flat_dfs read tests. by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2772 - fix(ci): do not overwrite last_log_file among tests by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2759 - feat(server): support cluster replication by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#2748 - fix: fiber preempts on read path and OnCbFinish() clears fetched_items\_ by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2763 - chore(ci): open last_log_file in append mode by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2776 - doc(README): fix outdated expiry ranges description by [@enjoy-binbin](https://togithub.com/enjoy-binbin) in [dragonflydb/dragonfly#2779 - Benchmark runner by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#2780 - chore(replication-tests): add cache_mode on test replication all by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2685 - feat(tiering): DiskStorage by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2770 - chore: add a boilerplate for bloom filter family by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2782 - chore: introduce conversion routines between JsonType and FlatJson by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2785 - chore: Fix memcached flags not updated by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2787 - chore: remove duplicate code from dash and simplify by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2765 - chore: disable test_cluster_slot_migration by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2788 - fix: new\[] delete\[] missmatch in disk_storage by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2792 - fix: sanitizers clang build and clean up some warnings by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2793 - chore: add bloom filter class by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2791 - chore: add SBF data structure by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2795 - chore(tiering): Disable compilation for MacOs by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2799 - chore: fix daily build by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2798 - chore: expose SBF via compact_object by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2797 - fix(ci): malloc trim on sanitizers workflow by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2794 - fix(cluster): Don't miss updates in FLUSHSLOTS by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2783 - feat: add bf.(m)add and bf.(m)exists commands by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2801 - fix: SBF memory leaks by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2803 - chore: refactor StringFamily::Set to use CmdArgParser by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2800 - fix: propagate memcached flags to replica by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2807 - DFLYMIGRATE ACK refactoring by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2790 - feat: add master lsn and journal_executed dcheck in replica via ping by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2778 - fix: correct json response for errors by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2813 - chore: bloom test - cover corner cases by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2806 - bug(server): do not write lsn opcode to journal by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#2814 - chore: Fix build by disabling the tests. by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2821 - fix(replication): replication with multi shard sync enabled lagging by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#2823 - fix: io_uring/fibers bug in DnsResolve by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2825 ##### New Contributors - [@manojks1999](https://togithub.com/manojks1999) made their first contribution in [dragonflydb/dragonfly#2659 - [@fafg](https://togithub.com/fafg) made their first contribution in [dragonflydb/dragonfly#2706 - [@enjoy-binbin](https://togithub.com/enjoy-binbin) made their first contribution in [dragonflydb/dragonfly#2779 ##### Huge thanks to all the contributors! ❤️ 🇮🇱 🇺🇦 **Full Changelog**: dragonflydb/dragonfly@v1.15.0...v1.16.0 </details>  Co-authored-by: repo-jeeves[bot] <106431701+repo-jeeves[bot]@users.noreply.github.com>

adiholden requested review from chakaz and dranikpg March 19, 2024 15:43

adiholden force-pushed the replicate_cluster branch from 569190c to dccf89c Compare March 19, 2024 20:55

chakaz reviewed Mar 20, 2024

View reviewed changes

dranikpg previously approved these changes Mar 20, 2024

View reviewed changes

adiholden dismissed dranikpg’s stale review via 2d82f10 March 21, 2024 20:28

adiholden added 7 commits March 24, 2024 21:37

feat(server): support cluster replication

b137851

Signed-off-by: adi_holden <adi@dragonflydb.io>

remove commented

cbd89f7

Signed-off-by: adi_holden <adi@dragonflydb.io>

fix build after rebase

d51540f

Signed-off-by: adi_holden <adi@dragonflydb.io>

fix build

70d9cc6

Signed-off-by: adi_holden <adi@dragonflydb.io>

fix loading state

dfa10d6

Signed-off-by: adi_holden <adi@dragonflydb.io>

fix CR

4fd712a

Signed-off-by: adi_holden <adi@dragonflydb.io>

flushslots get slot ranges

a017887

Signed-off-by: adi_holden <adi@dragonflydb.io>

adiholden force-pushed the replicate_cluster branch from 2d82f10 to a017887 Compare March 24, 2024 19:56

adiholden requested review from chakaz and dranikpg March 24, 2024 20:10

redis cluster takes more time to load on github

8ff8b63

Signed-off-by: adi_holden <adi@dragonflydb.io>

chakaz reviewed Mar 25, 2024

View reviewed changes

adiholden added 4 commits March 25, 2024 13:01

better replica args parser

4fe0635

Signed-off-by: adi_holden <adi@dragonflydb.io>

cleanup

52198aa

Signed-off-by: adi_holden <adi@dragonflydb.io>

add func to stop all cluster replicas

29b4f8f

Signed-off-by: adi_holden <adi@dragonflydb.io>

increase sleep time in test

a72e220

Signed-off-by: adi_holden <adi@dragonflydb.io>

adiholden requested a review from chakaz March 25, 2024 19:17

chakaz reviewed Mar 26, 2024

View reviewed changes

fix parser error check

2e43a15

Signed-off-by: adi_holden <adi@dragonflydb.io>

adiholden requested a review from chakaz March 26, 2024 11:17

chakaz approved these changes Mar 26, 2024

View reviewed changes

adiholden merged commit 2ad7439 into main Mar 26, 2024
10 checks passed

adiholden deleted the replicate_cluster branch March 26, 2024 13:26

	if (slot_id_end > ClusterConfig::kMaxSlotNum) {
	if ((slot_id_end > ClusterConfig::kMaxSlotNum) \|\| (slot_id_end < slot_id_start)) {

	absl::StrAppend(&slots_str, absl::StrCat(i, " "));
	absl::StrAppend(&slots_str, i, " ");

-    }
+    }
+    if (!slots_str.empty()) {
+      slots_str.pop_back();
+    }

	LOG(WARNING) << GlobalStateName(new_state) << " in progress, ignored";
	LOG(WARNING) << new_state << " in progress, ignored";

		GlobalState global_state_ = GlobalState::ACTIVE; // protected by mu_;
		uint32_t loading_state_counter_ = 0; // protected by mu_;

	return cntx->SendError("ADDREPLICAOF does not supprot no one");
	return cntx->SendError("ADDREPLICAOF does not support no one");

feat(server): support cluster replication #2748

feat(server): support cluster replication #2748

Conversation

adiholden commented Mar 19, 2024

romange commented Mar 19, 2024

chakaz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment