forked from apache/bookkeeper
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
BOOKKEEPER-946: Provide an option to delay auto recovery of lost bookies
If auto recovery is enabled, and a bookie goes down for upgrade or even if it looses zk connection intermittently, the auditor detects it as a lost bookie and starts under replication detection and the replication workers on other bookie nodes start replicating the under replicated ledgers. All of this stops once the bookie comes up but by then a few ledgers would get replicated. Given the fact that we have multiple copies of data, it is probably not necessary to start the recovery as soon as a bookie goes down. We can wait for an hour or so and then start recovery. This should cover cases like planned upgrade, intermittent network connectivity loss, etc. This change: 1) Provides a bookie option 'lostBookieRecoveryDelay' in secs, which when set to a non zero value, will delay the start of recovery by that number of seconds. By default, this option is set to 0; which means the audit is not delayed. 2) If another bookie were to go down in this interval, the recovery is immediately started and the one scheduled for future is canceled. 3) Adds counters to track how many audits were delayed(apache#1) and how many scheduled audits were canceled due to multiple bookie failures(apache#2). 4) Four junit tests to verify the new feature.
- Loading branch information
1 parent
9dc05fc
commit fd40702
Showing
5 changed files
with
350 additions
and
31 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.