The state of each server node is inconsistent

[wangyushuai]

Overview of the Issue

Node Az2-1 thinks that the state of node AZ2-0 is leaving, but the state of node AZ2-0 in the master node or other nodes except Az2-1 is alive. What is the reason for this? What should be done?

• Az2-0 query:

• But Az2-1 query:
  

Log Fragments

1. AZ2-0:

    2020/09/15 11:26:05 [ERR] yamux: Failed to write header: write tcp 10.0.0.9:8300->10.0.0.7:58123: use of closed network connection
    2020/09/22 01:09:54 [ERR] raft-net: Failed to flush response: write tcp 10.0.0.9:8300->10.0.0.11:60527: write: broken pipe

2. AZ2-1：

2020/09/10 19:22:16 [ERR] yamux: Failed to write header: write tcp 10.0.0.8:8300->10.0.0.13:41374: use of closed network connection
2020/09/22 01:09:55 [ERR] raft-net: Failed to flush response: write tcp 10.0.0.8:8300->10.0.0.11:38522: write: broken pipe
2020/09/27 10:48:14 [ERR] consul.rpc: multiplex conn accept failed: read tcp 10.0.0.8:8300->10.0.0.6:47987: read: connection reset by peer from=10.0.0.6:47987

3. AZ1-1 master node

    2020/09/22 01:09:54 [ERR] raft: Failed to AppendEntries to {Voter 67317135-7676-3372-7474-676f616f3231 10.0.0.8:8300}: read tcp 10.0.0.11:55225->10.0.0.8:8300: i/o timeout

[wangyushuai]

@mkeeler @johncowen @rboyer

[jsosulska]

Hi @wangyushuai

Thanks for posting. I don't have enough information to respond right now, so I would like to ask some questions.

• Can you please explain your architecture? I see there are a few nodes named similarly. Are there multiple NIC cards?
• Is there a reason you are running 1.4.4? This is a very old version
• Have you tried to restart the agent on that node?

[wangyushuai]

Hi @wangyushuai

Thanks for posting. I don't have enough information to respond right now, so I would like to ask some questions.

• Can you please explain your architecture? I see there are a few nodes named similarly. Are there multiple NIC cards?
• Is there a reason you are running 1.4.4? This is a very old version
• Have you tried to restart the agent on that node?

Hi @jsosulska , thanks for your reply

1. Each node in the screenshot is a server node. Each server node is in a different docker container. Az1-0 and Az2-0 nodes represent different zones server nodes;