Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: remove RAFT CF #4372

Closed
wants to merge 37 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
c7ea3df
store: upgrade data from v2 to v3
overvenus Feb 28, 2019
e79b3f3
raftstore/store: change bootstrap procedure
overvenus Mar 1, 2019
ce988a7
raftstore: partial change snapshot procedure
overvenus Mar 5, 2019
6c0d46d
server/debug: update raft and kv engine usage for debugger
overvenus Mar 6, 2019
d6e6ce8
raftstore: update raft and kv usage for consisitency check
overvenus Mar 6, 2019
ffcfac1
raftstrore: update raft and kv engine usage for apply
overvenus Mar 6, 2019
a9f17c8
*: remove remaining RAFT cf usage
overvenus Mar 7, 2019
2c7a4cc
*: deprecate RAFT cf config
overvenus Mar 7, 2019
450b217
*: add tests for upgrading
overvenus Mar 7, 2019
5537b87
*: add tests for bootstrapping
overvenus Mar 10, 2019
d2d6535
raftstore/store: take snapshots in apply worker
overvenus Mar 13, 2019
ef5c50c
Address lint
overvenus Mar 14, 2019
d5edb4b
Address lints and skip upgrade if it's a new node
overvenus Mar 14, 2019
c9de4c1
Address lints and update tests
overvenus Mar 14, 2019
45c1885
util: change crit to error
overvenus Mar 18, 2019
7319400
Merge branch 'master' into remove/raftcf
overvenus Mar 25, 2019
86559c9
Revert "util: change crit to error"
overvenus Mar 25, 2019
31ccd9d
Address fmt lints
overvenus Mar 25, 2019
6e5997d
apply: always sync wirte kv engine before updating raft apply state
overvenus Mar 28, 2019
971145d
apply: clear raft wb if there is no sync cmd
overvenus Apr 2, 2019
d0a6597
Merge commit '5a3ced50e' into remove/raftcf
overvenus Apr 13, 2019
c6d08ec
Merge branch 'master' into remove/raftcf
overvenus Apr 14, 2019
39789d9
tests: stable test_node_lease_unsafe_during_leader_transfers
overvenus Apr 15, 2019
202fe7e
Merge branch 'master' into remove/raftcf
overvenus Jun 22, 2019
0028f2e
Address comments
overvenus Apr 29, 2019
7bbe7c4
Merge branch 'master' into remove/raftcf
overvenus Apr 29, 2019
4992b1e
Address comments
overvenus Apr 29, 2019
edc2807
raftstore: ensure no pending snapshot requests before applying snapshots
overvenus Apr 29, 2019
9c8db23
Merge branch 'master' into remove/raftcf
overvenus May 7, 2019
6a21fbc
apply: gc raft write batch
overvenus May 7, 2019
b61f9b7
fix typo
overvenus May 7, 2019
9313942
Merge branch 'master' into remove/raftcf
overvenus May 7, 2019
836cca4
Merge branch 'master' into remove/raftcf
overvenus May 9, 2019
ab715ce
Merge branch 'master' into remove/raftcf
overvenus May 11, 2019
4d4d5d7
tests: cover update apply state unexpectedly
overvenus May 11, 2019
67e84b6
Merge branch 'master' into remove/raftcf
overvenus May 29, 2019
f835b8b
Merge branch 'master' into remove/raftcf
overvenus Jun 5, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 22 additions & 22 deletions src/raftstore/store/fsm/apply.rs
Original file line number Diff line number Diff line change
Expand Up @@ -419,11 +419,17 @@ impl ApplyContext {
// large raft write batch which causes high write duration.
// To keep raft write batch small we can clear it if there is no
// sync cmd in the current cmd batch.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious about that when the sync_log_hint is true, but we lost some apply states for some regions. If some of these regions are unlucky and always lost apply states, what will happen?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It restarts apply from the latest saved applied index, since all commands are idempotent, it will be consistent with other peers eventually.

//
// It is required to be cleared by the end of this function,
// otherwise it may write stall states, eg raftstore writes a apply
overvenus marked this conversation as resolved.
Show resolved Hide resolved
// state when applying snapshot which is newer then
overvenus marked this conversation as resolved.
Show resolved Hide resolved
// the apply state here.
//
// TODO: we need to flush apply states periodically to avoid
// re-apply too many entries after restart.
self.raft_wb_mut().clear();
}
if self.sync_log_hint {
self.sync_log_hint = false;
}
self.sync_log_hint = false;
for cbs in self.cbs.drain(..) {
cbs.invoke_all(&self.host);
}
Expand Down Expand Up @@ -476,14 +482,8 @@ impl ApplyContext {
pub fn flush(&mut self) {
// TODO: this check is too hacky, need to be more verbose and less buggy.
let t = match self.timer.take() {
Some(t) => Some(t),
None => {
if self.sync_log_hint {
None
} else {
return;
}
}
Some(t) => t,
None => return,
};

// Write to engine
Expand All @@ -504,15 +504,13 @@ impl ApplyContext {
}
}

if let Some(t) = t {
STORE_APPLY_LOG_HISTOGRAM.observe(duration_to_sec(t.elapsed()) as f64);
slow_log!(
t,
"{} handle ready {} committed entries",
self.tag,
self.committed_count
);
}
STORE_APPLY_LOG_HISTOGRAM.observe(duration_to_sec(t.elapsed()) as f64);
slow_log!(
t,
"{} handle ready {} committed entries",
self.tag,
self.committed_count
);
self.committed_count = 0;
}
}
Expand Down Expand Up @@ -2610,8 +2608,10 @@ impl ApplyFsm {
if self.delegate.pending_remove || self.delegate.stopped {
return;
}
if apply_ctx.timer.is_none() {
apply_ctx.timer = Some(SlowTimer::new());
}

let region_id = self.delegate.id();
if apply_ctx.raft_wb.is_none() {
apply_ctx.raft_wb = Some(WriteBatch::with_capacity(DEFAULT_RAFT_WB_SIZE));
}
Expand All @@ -2627,7 +2627,7 @@ impl ApplyFsm {
error!(
"schedule snapshot failed";
"error" => ?e,
"region_id" => region_id,
"region_id" => self.delegate.region_id(),
"peer_id" => self.delegate.id()
);
}
Expand Down