fix(server): saving is not a server state #2613

adiholden · 2024-02-18T19:50:17Z

fixes #2597
The bug:
server crashed because when replicaof no one was executed the replica was creating a snapshot and there for the server state was not active but saving.
check fail -server_family.cc:2142] Check failed: service_.SwitchState(GlobalState::LOADING, GlobalState::ACTIVE).first == GlobalState::ACTIVE Server is set to replica no one, yet state is not active!
The fix:

remove saving from server states
move save stage controller to be a member of server family

Signed-off-by: adi_holden <adi@dragonflydb.io>

romange · 2024-02-18T19:52:26Z

src/server/detail/save_stages_controller.cc

@@ -264,6 +256,10 @@ void SaveStagesController::SaveRdb() {
  trans_->ScheduleSingleHop(std::move(cb));
 }

+uint32_t SaveStagesController::GetCurrentSaveDuratio() {


GetCurrentSaveDuration

romange · 2024-02-18T19:59:28Z

src/server/server_family.cc

-                          StrCat(GlobalStateName(new_state), " - can not save database")};
-    }
+  auto state = service_.GetGlobalState();
+  if (!ignore_state && (state != GlobalState::ACTIVE)) {


please add comment explaining the context for this check and for ignore_state variable.
i know that you did not add ignore_state, but curious to understand why we need this check.

romange · 2024-02-18T20:02:19Z

src/server/detail/save_stages_controller.h

@@ -19,16 +17,28 @@ class Service;

 namespace detail {

+class SnapshotStorage;
+
+struct LastSaveInfo {


nit: since we return it from a ServerFamily function, it can be declared outside detail namespace

romange · 2024-02-18T20:10:07Z

src/server/server_family.h


  // Load snapshot from file (.rdb file or summary.dfs file) and return
  // future with error_code.
  util::fb2::Future<GenericError> Load(const std::string& file_name);

  bool IsSaving() const {
-    return is_saving_.load(std::memory_order_relaxed);
+    std::lock_guard lk(save_mu_);
+    return save_controller_ && save_controller_->IsSaving();


seems that this function is now used only for tests.
I think that returning bool(save_controller_) is enough for this

actually .. I debuged the test for a while to understand that I do need it, as it takes some time for the save executed command to be scheduled in the transaction queue. If I dont have this is_saving_ set after the scheduling the test SaveFlush fails as it will schedule flushall before the save

Let's not complicate real code for testing reasons (especially without comments or TEST prefix), we have lot's of other ways to check if saving is in progress. For example check tl_slice_snapshots, which should be easiest. We can also check transaction metrics

this change was not introduced in this PR. is_saving_ was set there before. I change the logic there

romange · 2024-02-18T20:10:57Z

src/server/detail/save_stages_controller.h

 private:
  absl::Time start_time_;
  std::filesystem::path full_path_;
  bool is_cloud_;
+  bool is_saving_ = false;


i think it's not really needed. see my comment below

dranikpg · 2024-02-18T20:43:13Z

src/server/server_family.h


  // Load snapshot from file (.rdb file or summary.dfs file) and return
  // future with error_code.
  util::fb2::Future<GenericError> Load(const std::string& file_name);

  bool IsSaving() const {
-    return is_saving_.load(std::memory_order_relaxed);
+    std::lock_guard lk(save_mu_);
+    return save_controller_ && save_controller_->IsSaving();


Let's not complicate real code for testing reasons (especially without comments or TEST prefix), we have lot's of other ways to check if saving is in progress. For example check tl_slice_snapshots, which should be easiest. We can also check transaction metrics

dranikpg · 2024-02-18T20:46:52Z

tests/dragonfly/snapshot_test.py

@@ -125,21 +125,21 @@ async def test_dbfilenames(

 @pytest.mark.slow
 @dfly_args({**BASIC_ARGS, "dbfilename": "test-cron", "snapshot_cron": "* * * * *"})
-async def test_cron_snapshot(tmp_path: Path, async_client: aioredis.Redis):
+async def test_cron_snapshot(tmp_dir: Path, async_client: aioredis.Redis):


my bad 😅 ... but why did it pass 🤔

it did not. the slow tests are not running in regression tests

dranikpg · 2024-02-18T20:50:21Z

src/server/detail/save_stages_controller.h

  LastSaveInfo* last_save_info_ ABSL_GUARDED_BY(save_mu_);
  util::fb2::Mutex* save_mu_;


Just a though I realize now (and not while extracting this class), it'd be more convenient if returned LastSaveInfo just by value from Save() instead of passing a mutex by pointer

dranikpg · 2024-02-18T20:53:09Z

tests/dragonfly/replication_test.py

+    save_task = asyncio.create_task(save_replica())
+    await asyncio.sleep(0.1)  # let save start
+    await c_replica.execute_command("replicaof no one")
+    await save_task


Once we assert replicaof no one returns OK and doesn't crash, the test is over. We can kill the replica and cancel the save task (it'll be knocked out with an error either way)

Maybe we can busy loop on saving_metric > 0 to start quicker? Not to save time here, but to be able to use much less than 100k keys (we rarely use such large values like 100k in pytests + debug). The snapshot starting shouldn't take more than a few milliseconds on a slow machine, and a slow machine won't be quick enough to save even 10k large values until the replicaof arrives immediately after

I updated the test to use 10k keys and a busy loop. I do want to want to wait for the save to finish and not just exit after issue replicaof no one, this test will check that saving will finish after promotion to master and not only try to reproduce a specific crash we had

Signed-off-by: adi_holden <adi@dragonflydb.io>

dranikpg · 2024-02-19T11:57:40Z

src/server/server_family.h

+  bool TEST_IsSaving() const {
+    std::lock_guard lk(save_mu_);
+    return save_controller_ && save_controller_->TEST_IsSaving();
  }


This is obviously not safe and that's why the test segfaults. I meant to check thread local snapshots not tied to any saver instance. Either way, I don't think it's that important, no matter how it works, we can clean up later 😅

Signed-off-by: adi_holden <adi@dragonflydb.io>

* fix(server): saving is not a server state Signed-off-by: adi_holden <adi@dragonflydb.io>

fix(server): saving is not a server state

3fa03f3

Signed-off-by: adi_holden <adi@dragonflydb.io>

adiholden requested review from dranikpg, BorysTheDev and romange February 18, 2024 19:50

add pytest that reproduces the crash

00c7f18

Signed-off-by: adi_holden <adi@dragonflydb.io>

romange reviewed Feb 18, 2024

View reviewed changes

dranikpg reviewed Feb 18, 2024

View reviewed changes

fix PR comments

bcad69d

Signed-off-by: adi_holden <adi@dragonflydb.io>

romange previously approved these changes Feb 19, 2024

View reviewed changes

dranikpg reviewed Feb 19, 2024

View reviewed changes

improve pytest

9790c32

Signed-off-by: adi_holden <adi@dragonflydb.io>

adiholden dismissed romange’s stale review via 9790c32 February 19, 2024 12:00

fix is saving

47e1d92

Signed-off-by: adi_holden <adi@dragonflydb.io>

dranikpg approved these changes Feb 19, 2024

View reviewed changes

adiholden enabled auto-merge (squash) February 19, 2024 15:16

adiholden merged commit 15b3fb1 into main Feb 19, 2024
10 checks passed

adiholden deleted the fix_2597 branch February 19, 2024 15:20

adiholden added a commit that referenced this pull request Feb 20, 2024

fix(server): saving is not a server state (#2613)

6cf3e24

* fix(server): saving is not a server state Signed-off-by: adi_holden <adi@dragonflydb.io>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(server): saving is not a server state #2613

fix(server): saving is not a server state #2613

adiholden commented Feb 18, 2024

romange Feb 18, 2024

romange Feb 18, 2024

romange Feb 18, 2024

romange Feb 18, 2024

adiholden Feb 18, 2024

dranikpg Feb 18, 2024

adiholden Feb 19, 2024

romange Feb 18, 2024

dranikpg Feb 18, 2024

dranikpg Feb 18, 2024

adiholden Feb 19, 2024

dranikpg Feb 18, 2024

dranikpg Feb 18, 2024

dranikpg Feb 18, 2024 •

edited

adiholden Feb 19, 2024

dranikpg Feb 19, 2024

		LastSaveInfo* last_save_info_ ABSL_GUARDED_BY(save_mu_);
		util::fb2::Mutex* save_mu_;

fix(server): saving is not a server state #2613

fix(server): saving is not a server state #2613

Conversation

adiholden commented Feb 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dranikpg Feb 18, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dranikpg Feb 18, 2024 •

edited