Skip to content

Commit

Permalink
Fix reopen voting logic in db_stress to prevent hangs (facebook#5876)
Browse files Browse the repository at this point in the history
Summary:
When multiple operations are performed in a db_stress thread in one loop
iteration, the reopen voting logic needs to take that into account. It
was doing that for MultiGet, but a new option was introduced recently to
do multiple iterator seeks per iteration, which broke it again. Fix the
logic to be more robust and agnostic of the type of operation performed.
Pull Request resolved: facebook#5876

Test Plan: Run db_stress

Differential Revision: D17733590

Pulled By: anand1976

fbshipit-source-id: 787f01abefa1e83bba43e0b4f4abb26699b2089e
  • Loading branch information
anand76 authored and facebook-github-bot committed Oct 3, 2019
1 parent 155d376 commit 8d64abe
Showing 1 changed file with 8 additions and 7 deletions.
15 changes: 8 additions & 7 deletions tools/db_stress.cc
Original file line number Diff line number Diff line change
Expand Up @@ -2086,18 +2086,19 @@ class StressTest {
const int delBound = writeBound + (int)FLAGS_delpercent;
const int delRangeBound = delBound + (int)FLAGS_delrangepercent;
const uint64_t ops_per_open = FLAGS_ops_per_thread / (FLAGS_reopen + 1);
int multiget_batch_size = 0;

thread->stats.Start();
for (uint64_t i = 0; i < FLAGS_ops_per_thread; i++) {
for (uint64_t i = 0, prev_i = 0; i < FLAGS_ops_per_thread; i++) {
if (thread->shared->HasVerificationFailedYet()) {
break;
}
// Check if the multiget batch crossed the ops_per_open boundary. If it
// did, then we should vote to reopen
// In case i is incremented more than once due to multiple operations,
// such as MultiGet or iterator seeks, check whether we have crossed
// the ops_per_open boundary in the previous iteration. If it did,
// then vote to reopen
if (i != 0 &&
(i % ops_per_open == 0 ||
i % ops_per_open < (i - multiget_batch_size) % ops_per_open)) {
i % ops_per_open < prev_i % ops_per_open)) {
{
thread->stats.FinishedSingleOp();
MutexLock l(thread->shared->GetMutex());
Expand All @@ -2118,6 +2119,7 @@ class StressTest {
// thread->stats.Start();
}
}
prev_i = i;

// Change Options
if (FLAGS_set_options_one_in > 0 &&
Expand Down Expand Up @@ -2312,14 +2314,13 @@ class StressTest {
// Reset this in case we pick something other than a read op. We don't
// want to use a stale value when deciding at the beginning of the loop
// whether to vote to reopen
multiget_batch_size = 0;
if (prob_op >= 0 && prob_op < (int)FLAGS_readpercent) {
// OPERATION read
if (FLAGS_use_multiget) {
// Leave room for one more iteration of the loop with a single key
// batch. This is to ensure that each thread does exactly the same
// number of ops
multiget_batch_size = static_cast<int>(
int multiget_batch_size = static_cast<int>(
std::min(static_cast<uint64_t>(thread->rand.Uniform(64)),
FLAGS_ops_per_thread - i - 1));
// If its the last iteration, ensure that multiget_batch_size is 1
Expand Down

0 comments on commit 8d64abe

Please sign in to comment.