-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add a router tag for all mutations after enabling HA
We found a data corruption bug when switching from a single region to two regions, i.e., re-enabling HA. The exact sequence of corruption for the test is: 1. Epoch 6: 2 regions 2. Epoch 8: change usable_region to 1, txn commit at version V. 3. Epoch 10: usable_region is now 2, recoverAt is V. and V is copied to the newly recruited tlogs. Remote SS peeked the tlog, but didn't persist the V's data yet. 4. Restart. Epoch 12, another recovery 5. Epoch 14: remote tlog starts at Unrecovered < V , actually Unrecovered == Epoch 8's endVersion. So pullAsyncData is pulling with log router tag, and mutations at V don't have router tags. To reproduce: -f ./tests/restarting/from_7.3.0/ConfigureTestRestart-1.toml -b on -s 1855375089 -f ./tests/restarting/from_7.3.0/ConfigureTestRestart-2.toml -b on -s 1855375090 --restarting commit d24a62c, clang build So the problem is with the tlog data copied from epoch 8 to 10 can be lost, because they don't have a log router tag. So at epoch 14, when pullAsyncData tries to copy the data from version V, they are not copied. The solution is to add a static router tag (-2, 0) to all mutations after HA is enabled. Since this transaction will trigger a recovery, the next epoch has log router tags, so the copied range will have the proper tag to be pulled from remote side, i.e., log routers. For the tlog data not copied from the old epoch when usable_region is 1, remote side storage servers will peek from old tlogs.
- Loading branch information
Showing
5 changed files
with
16 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters