Master service fails when orchestrator is down #203
Description
The rebooted node failed with the following message:
New IP of rebooted node: 10.233.5.244
Old IP before rebooted: 10.233.5.243
Other nodes without rebooted: 10.233.0.160 and 10.233.17.75
- Rebooted node side
2019/01/15 04:52:56 [INFO] raft: Node at 10.233.5.244:10008 [Follower] entering Follower state (Leader: "")
2019/01/15 04:52:58 [WARN] raft: Heartbeat timeout from "" reached, starting election
2019/01/15 04:52:58 [INFO] raft: Node at 10.233.5.244:10008 [Candidate] entering Candidate state
2019/01/15 04:52:58 [WARN] raft: Remote peer 10.233.0.160:10008 does not have local node 10.233.5.244:10008 as a peer
2019/01/15 04:52:58 [WARN] raft: Remote peer 10.233.17.75:10008 does not have local node 10.233.5.244:10008 as a peer
- existed nodes side
019/01/15 04:55:55 [DEBUG] raft: Failed to contact 10.233.5.243:10008 in 3m59.07956023s
2019/01/15 04:55:56 [DEBUG] raft: Failed to contact 10.233.5.243:10008 in 3m59.528575654s
2019/01/15 04:55:56 [WARN] raft: Rejecting vote request from 10.233.5.244:10008 since we have a leader: 10.233.17.75:10008
The rebooted node failed to join the existed cluster because
- the ip of rebooted node had changed
- other nodes keep the old ip of rebooted node