brain split happened when network interrupts between dc #825
Comments
|
what's the command line to start the 7 master servers? |
|
7 master start commad like this : |
|
what kind of network interruption? It'll be good if this can be reproduced. basically the isolated dc3 elected one leader from the 2 masters, even though the quorum is 4. |
|
dc3 cannot reach other dc beacuse of network provider's line interruption, but dc3 internal network is ok。 totally 7 nodes , the isolated dc3 elected one leader from the 2 dc3's masters , not reach half of 7 nodes. i think it can be reproduced by iptables to interrupt the network between dc3 and other dc |
|
It'll be good to see the actual logs, if you still have them. |
|
about 00:42 network faliure happened , Back to normal an hour later 。 the second master seems be elected when network goes well after an hour
4.dc1 another master node: 5.dc1 third master node: |
|
Sounds like a critical bug easy to reproduce. |
|
Maybe using this tried and true raft implementation would be a solution? https://github.com/etcd-io/etcd/tree/master/raft |
|
How to reproduce it and prove either way?
…On Wed, Jan 16, 2019 at 12:14 AM ingardm ***@***.***> wrote:
Maybe using this tried and true raft implementation would be a solution?
https://github.com/etcd-io/etcd/tree/master/raft
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#825 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABeL778mNwJboENFJIFCbGhcFmCzbN8hks5vDt9SgaJpZM4Z4u4C>
.
|
|
Reproduce as below:
now master-0 is leader, master-1 and master-2 are followers. Then execute the following steps:
I have also tested removing go keyword in topology.go#L94, it will solve the problem above, but bring the side effect that
Updated:
|
|
thanks! this seems a timeout issue where node-0 needs to actively detect stream gRPC disconnections. |
|
@PapaYofen please check whether this 1d103e3 fix the problem. |
It only reduces the append entry request time to follower, but can't solve split-brain problem.
2 and 3 need updating raft codes which i do not understand thoroughly, maybe etcd raft is a better choice solving this split-brain issue already , and also implementing preVote algorithm to reduce disturb when minority part connects again to majority part. |
|
I think it is fixed. Let me know if otherwise. |
|
@chrislusf @PapaYofen master m3 log:Now block dc3 and make it to be an isolated network environment, v3 disconnect from m1 and connect to m3 , v1 and v2 also connect to m1。m1 and m3 ,both nodes are leader? master m3 log：dc3 network is back to normal , m3 to be the all cluster leader filer f1 netstat: filer also connect to preleader and current leader |
|
added an idle timeout. Not sure how to make this unit testable. Please help to test this. |
|
|
what's your command to do this? |
|
sudo iptables -I INPUT -s node-1-ip -j DROP sudo iptables -I INPUT -s node-2-ip -j DROP |
|
PapaYofen/raft@ecfccfc maybe this two commits can fix split-brain problem |
I tried ,but also two leaders.
|
@PapaYofen @chrislusf the latest commits seems work fine , the leader will be elected from more than half of nodes cluster , the leader from less than half of nodes cluster will become candidate , There will be always one leader node |
|
if chirs approves this solution, i will pull a merge request |
@chrislusf how about this ,expect for the request |
|
@accumulatepig I am not sure what are the exact versions of code from the seaweedfs and raft repos you have tested. Besides, I have changed the raft code from http to grpc recently, so the change does not apply any more. |
|
About raft:
About seaweedfs: func (t *Topology) NextVolumeId() storage.VolumeId {
vid := t.GetMaxVolumeId()
next := vid.Next()
go t.RaftServer.Do(NewMaxVolumeIdCommand(next))
return next
}to func (t *Topology) NextVolumeId() (storage.VolumeId, error) {
vid := t.GetMaxVolumeId()
next := vid.Next()
if _, err := t.RaftServer.Do(NewMaxVolumeIdCommand(next)); err != nil {
return 0, err
}
return next, nil
} |
|
This commit 7a493bb may be the real fix. Setting up the test env is too much work, I have not checked it yet. |
i dont think this commit fix brain split issue, due to raft not support auto-changing role from leader to candidate after brain split. |
@chrislusf yes ,I tried commit 7a493bb ,but it doesn't work , same as @PapaYofen said ,the leader not support change role from leader to candidate ,and the sceond leader elected when brain split happened |
|
@accumulatepig thanks for testing and confirming |
|
raft pr has also been sent |
|
@chrislusf @PapaYofen |
|
thanks for your test, i have found the root cause |
weed vesion:1.15
dc1 and dc2 is ok , so dc3 is a network isolated island 。
The text was updated successfully, but these errors were encountered: