Permalink
Browse files

[Backport] MB-4595 Schedule backfill for a fresh empty client

There is a case that causes data loss after rebalance in
the cluster running for a very short period:

1) Start the one node cluster and load a small number of items
into the node. The node will still have the open checkpoint with
id 1 for each vbucket.
2) Shutdown the node and start it again. After warmup, the node
still has the open checkpoint 1 for each vbucket, but each
checkpoint doesn't have any items in its datastructure.
3) Add another fresh node to the cluster and rebalane in. In this
case, no backfill tasks are scheduled for vbucket takeovers
because the fresh node starts with the open checkpoint id 1 and
the original node still has the open checkpoint id 1.

Consequently, each vbucket takeover is completed without backfill
but does not send any items from the open checkpoint 1 because
it doesn't have any items. This causes data loss after rebalance.

To resolve the above issue, we always schedule the backfill for
the fresh client with empty data (i.e., checkpoint id 1).
The better and more comprehensive solution would be to restore the
open checkpoint as part of warmup, which we will provide soon.

Change-Id: I4a7bc1ad400e6406ff2f900749514f2cabdc90c9
Reviewed-on: http://review.couchbase.org/14223
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.couchbase.org/15670
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
  • Loading branch information...
1 parent ed4da2a commit 1a51cc07ffca8e7f9c10c4e8be04102e85acaa76 @chiyoung chiyoung committed Mar 22, 2012
Showing with 19 additions and 4 deletions.
  1. +4 −0 checkpoint.cc
  2. +2 −0 checkpoint.hh
  3. +1 −0 ep_testsuite.cc
  4. +12 −4 tapconnection.cc
View
@@ -509,6 +509,10 @@ std::list<std::string> CheckpointManager::getTAPCursorNames() {
return cursor_names;
}
+bool CheckpointManager::tapCursorExists(const std::string &name) {
+ return tapCursors.find(name) != tapCursors.end();
+}
+
bool CheckpointManager::isCheckpointCreationForHighMemUsage(const RCPtr<VBucket> &vbucket) {
bool forceCreation = false;
double current = static_cast<double>(stats.currentSize.get() + stats.memOverhead.get());
View
@@ -324,6 +324,8 @@ public:
std::list<std::string> getTAPCursorNames();
+ bool tapCursorExists(const std::string &name);
+
/**
* Start onlineupdate - stop persisting mutation to disk
* @return :
View
@@ -3076,6 +3076,7 @@ static enum test_result test_tap_takeover(ENGINE_HANDLE *h, ENGINE_HANDLE_V1 *h1
case TAP_PAUSE:
testHarness.waitfor_cookie(cookie);
break;
+ case TAP_OPAQUE:
case TAP_NOOP:
break;
case TAP_MUTATION:
View
@@ -382,11 +382,19 @@ void TapProducer::registerTAPCursor(std::map<uint16_t, uint64_t> &lastCheckpoint
continue;
}
+ uint64_t chk_id_to_start = tapCheckpointState[vbid].currentCheckpointId;
+ // If this tap connection is for a new client with checkpoint 1, we should always
+ // schedule backfill because the tap producer could be restarted with the open
+ // checkpoint 1, but not restore the items in the open checkpoint.
+ bool empty_client =
+ !vb->checkpointManager.tapCursorExists(name) && (chk_id_to_start == 1);
// Check if the unified queue contains the checkpoint to start with.
- if(vb && !vb->checkpointManager.registerTAPCursor(name,
- tapCheckpointState[vbid].currentCheckpointId,
- closedCheckpointOnly, registeredTAPClient)) {
- if (backfillAge < current_time) { // Backfill is required.
+ bool chk_exists = vb->checkpointManager.registerTAPCursor(name,
+ chk_id_to_start,
+ closedCheckpointOnly,
+ registeredTAPClient);
+ if(!chk_exists || empty_client) {
+ if (backfillAge < current_time) {
TapCheckpointState st(vbid, 0, backfill);
tapCheckpointState[vbid] = st;
// As we set the cursor to the beginning of the open checkpoint when backfill

0 comments on commit 1a51cc0

Please sign in to comment.