Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

raftstore: reject transfer leader to the recently added peers #3878

Merged
merged 16 commits into from Dec 7, 2018

Conversation

Projects
None yet
8 participants
@nolouch
Copy link
Contributor

commented Dec 5, 2018

What have you changed? (mandatory)

The apply state not consistent with the leader, transfer leader to newly added peer probable failed . reject transfer leader to the recently added peers to prevent transferring failed.
This is a quick fix, need remove after pingcap/raft-rs#144 sloved.
ref: #3819

What are the type of the changes? (mandatory)

  • Improvemen

How has this PR been tested? (mandatory)

unit test.

manua test:
master(before) vs bransh(after):
prepare: add a special hot scheduler that move hot region and transfer the leader to the newly added hot peer. run oltp test.
result:
deepinscreenshot_select-area_20181206131739

Show resolved Hide resolved src/raftstore/store/peer.rs Outdated

nolouch added some commits Dec 4, 2018

raftstore: reject transfer leader to recently added peer
Signed-off-by: nolouch <nolouch@gmail.com>
add tests
Signed-off-by: nolouch <nolouch@gmail.com>

@nolouch nolouch force-pushed the nolouch:saft-transfer branch 2 times, most recently from 41cc231 to d87203e Dec 5, 2018

address
Signed-off-by: nolouch <nolouch@gmail.com>

@nolouch nolouch force-pushed the nolouch:saft-transfer branch from d87203e to f4ac4f6 Dec 5, 2018

Show resolved Hide resolved src/raftstore/store/config.rs Outdated
Show resolved Hide resolved src/raftstore/store/config.rs Outdated
Show resolved Hide resolved src/raftstore/store/peer.rs Outdated
Show resolved Hide resolved src/raftstore/store/peer.rs Outdated
@siddontang

This comment has been minimized.

Copy link
Contributor

commented Dec 5, 2018

em, can the leader know the applied index of the followers so that we can do a more accurate control?

@nolouch

This comment has been minimized.

Copy link
Contributor Author

commented Dec 5, 2018

@siddontang Currently, the apply index can only be known by itself. If need precise control, should change it in the raft library. This is a quick fix, and the long-term solution maybe is this.

@ngaut

This comment has been minimized.

Copy link
Member

commented Dec 5, 2018

I prefer to do a quick fix for now. And make it better later.

@rleungx rleungx added the P: Critical label Dec 5, 2018

nolouch added some commits Dec 5, 2018

address comments
Signed-off-by: nolouch <nolouch@gmail.com>
address
Signed-off-by: nolouch <nolouch@gmail.com>
fix
Signed-off-by: nolouch <nolouch@gmail.com>

@nolouch nolouch force-pushed the nolouch:saft-transfer branch from d15a8ea to 554cc21 Dec 5, 2018

@nolouch nolouch added the C: Raft label Dec 5, 2018

@siddontang

This comment has been minimized.

Copy link
Contributor

commented Dec 5, 2018

This still has the risk that the learner drops the transfer leader message if the learner applies too slow.

/cc @BusyJay

@BusyJay

This comment has been minimized.

Copy link
Contributor

commented Dec 5, 2018

Yes, it does. But it should be hardly possible.

nolouch added some commits Dec 6, 2018

reduce memery use
Signed-off-by: nolouch <nolouch@gmail.com>
address comment
Signed-off-by: nolouch <nolouch@gmail.com>

@nolouch nolouch force-pushed the nolouch:saft-transfer branch from a30ecb5 to c958715 Dec 6, 2018

@ngaut

This comment has been minimized.

Copy link
Member

commented Dec 6, 2018

Good job.

@nolouch

This comment has been minimized.

Copy link
Contributor Author

commented Dec 6, 2018

address comments
Signed-off-by: nolouch <nolouch@gmail.com>
@@ -893,6 +893,10 @@ impl<T: Transport, C: PdClient> Store<T, C> {
if p.is_leader() {
p.peers_start_pending_time.push((id, now));
}
// Add peer or promote the learner
if !peer.get_is_learner() {

This comment has been minimized.

Copy link
@BusyJay

BusyJay Dec 6, 2018

Contributor

I think only leader needs to do this.

This comment has been minimized.

Copy link
@nolouch

nolouch Dec 6, 2018

Author Contributor

Maybe transfer From A to B then to Cnew.

Show resolved Hide resolved src/raftstore/store/peer.rs Outdated
Show resolved Hide resolved src/raftstore/store/peer.rs Outdated

nolouch added some commits Dec 6, 2018

address comments
Signed-off-by: nolouch <nolouch@gmail.com>
@zhangjinpeng1987
Copy link
Member

left a comment

LGTM

Show resolved Hide resolved src/raftstore/store/peer.rs Outdated
Show resolved Hide resolved tests/integrations/raftstore/test_conf_change.rs Outdated
Show resolved Hide resolved tests/integrations/raftstore/test_conf_change.rs Outdated
address comments
Signed-off-by: nolouch <nolouch@gmail.com>
more stability
Signed-off-by: nolouch <nolouch@gmail.com>

@nolouch nolouch force-pushed the nolouch:saft-transfer branch from 962695f to 95890c2 Dec 7, 2018

@BusyJay

BusyJay approved these changes Dec 7, 2018

@zhangjinpeng1987
Copy link
Member

left a comment

LGTM

@zhangjinpeng1987 zhangjinpeng1987 merged commit f672088 into tikv:master Dec 7, 2018

3 checks passed

DCO All commits are signed off!
Details
ci/circleci: test Your tests passed on CircleCI!
Details
jenkins-ci-tikv/build Jenkins job succeeded.
Details

nolouch added a commit to nolouch/tikv that referenced this pull request Dec 8, 2018

nolouch added a commit to nolouch/tikv that referenced this pull request Dec 8, 2018

nolouch added a commit that referenced this pull request Dec 8, 2018

@nolouch nolouch deleted the nolouch:saft-transfer branch Dec 12, 2018

nolouch added a commit to nolouch/tikv that referenced this pull request Dec 14, 2018

nolouch added a commit to nolouch/tikv that referenced this pull request Dec 14, 2018

rleungx added a commit that referenced this pull request Dec 14, 2018

raftstore: reject transfer leader to the recently added peers (#3878) (
…#3929)

* raftstore: reject transfer leader to the recently added peers (#3878)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.