BlockchainReactor Timeout error while syncing

[melekes]

E[10-19|04:55:31.278] Stopping peer for error                      module=p2p peer="Peer{MConn{146.185.173.132:46656} 55AF537EBCFA out}" err="BlockchainReactor Timeout"
E[10-19|04:55:32.408] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:55:33.198] Failed to add peer                           module=p2p address=178.62.46.14:46656 err="Duplicate peer"
E[10-19|04:55:35.656] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:55:36.735] Failed to add peer                           module=p2p address=178.62.46.14:46656 err="Duplicate peer"
E[10-19|04:55:38.334] Stopping peer for error                      module=p2p peer="Peer{MConn{178.62.46.14:46656} B14916FAF38A out}" err="BlockchainReactor Timeout"
E[10-19|04:55:38.904] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:55:39.364] Stopping peer for error                      module=p2p peer="Peer{MConn{104.131.86.166:46656} 7A401C816E0C out}" err="BlockchainReactor Timeout"
E[10-19|04:55:40.276] Failed to add peer                           module=p2p address=178.62.46.14:46656 err="Duplicate peer"
E[10-19|04:55:41.723] Stopping peer for error                      module=p2p peer="Peer{MConn{146.185.173.132:46656} 55AF537EBCFA out}" err="BlockchainReactor Timeout"
E[10-19|04:55:42.022] Stopping peer for error                      module=p2p peer="Peer{MConn{104.131.86.166:46656} 7A401C816E0C out}" err="BlockchainReactor Timeout"
E[10-19|04:55:42.150] Stopping peer for error                      module=p2p peer="Peer{MConn{178.62.46.14:46656} B14916FAF38A out}" err="BlockchainReactor Timeout"
E[10-19|04:55:42.274] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:55:43.812] Failed to add peer                           module=p2p address=178.62.46.14:46656 err="Duplicate peer"
E[10-19|04:55:45.522] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:55:47.391] Failed to add peer                           module=p2p address=178.62.46.14:46656 err="Duplicate peer"
E[10-19|04:55:48.769] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:55:50.931] Failed to add peer                           module=p2p address=178.62.46.14:46656 err="Duplicate peer"
E[10-19|04:55:52.016] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:55:54.465] Failed to add peer                           module=p2p address=178.62.46.14:46656 err="Duplicate peer"
E[10-19|04:55:55.264] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:55:58.006] Failed to add peer                           module=p2p address=178.62.46.14:46656 err="Duplicate peer"
E[10-19|04:55:58.509] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:56:01.542] Failed to add peer                           module=p2p address=178.62.46.14:46656 err="Duplicate peer"
E[10-19|04:56:01.755] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:56:05.003] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:56:05.082] Failed to add peer                           module=p2p address=178.62.46.14:46656 err="Duplicate peer"
E[10-19|04:56:08.249] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:56:08.622] Failed to add peer                           module=p2p address=178.62.46.14:46656 err="Duplicate peer"
E[10-19|04:56:11.494] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:56:12.159] Failed to add peer                           module=p2p address=178.62.46.14:46656 err="Duplicate peer"
E[10-19|04:56:14.744] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:56:15.700] Failed to add peer                           module=p2p address=178.62.46.14:46656 err="Duplicate peer"
E[10-19|04:56:17.992] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:56:19.238] Failed to add peer                           module=p2p address=178.62.46.14:46656 err="Duplicate peer"
E[10-19|04:56:21.238] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:56:24.484] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"
E[10-19|04:56:27.730] Failed to add peer                           module=p2p address=104.131.86.166:46656 err="Duplicate peer"

[melekes]

E[10-19|04:52:38.969] Stopping peer for error                      module=p2p peer="Peer{MConn{178.62.46.14:46656} B14916FAF38A out}" err="Error: runtime error: invalid memory address or nil pointer dereference\nStack: goroutine 529485 [running]:\nruntime/debug.Stack(0xc4355cfb38, 0xb463e0, 0x11b1c30)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0xa7\ngithub.com/cosmos/gaia/vendor/github.com/tendermint/tendermint/p2p.(*MConnection)._recover(0xc439a28870)\n\t/home/ubuntu/go/src/github.com/cosmos/gaia/vendor/github.com/tendermint/tendermint/p2p/connection.go:206 +0x6e\npanic(0xb463e0, 0x11b1c30)\n\t/usr/local/go/src/runtime/panic.go:491 +0x283\ngithub.com/cosmos/gaia/vendor/github.com/tendermint/tendermint/blockchain.(*bpPeer).decrPending(0x0, 0x381)\n\t/home/ubuntu/go/src/github.com/cosmos/gaia/vendor/github.com/tendermint/tendermint/blockchain/pool.go:376 +0x22\ngithub.com/cosmos/gaia/vendor/github.com/tendermint/tendermint/blockchain.(*BlockPool).AddBlock(0xc4200e4000, 0xc4266d1f00, 0x40, 0xc432ac9640, 0x381)\n\t/home/ubuntu/go/src/github.com/cosmos/gaia/vendor/github.com/tendermint/tendermint/blockchain/pool.go:215 +0x139\ngithub.com/cosmos/gaia/vendor/github.com/tendermint/tendermint/blockchain.(*BlockchainReactor).Receive(0xc42050a780, 0xc420257740, 0x1171be0, 0xc42ff302d0, 0xc4384b2000, 0x381, 0x1000)\n\t/home/ubuntu/go/src/github.com/cosmos/gaia/vendor/github.com/tendermint/tendermint/blockchain/reactor.go:160 +0x712\ngithub.com/cosmos/gaia/vendor/github.com/tendermint/tendermint/p2p.createMConnection.func1(0x11e5040, 0xc4384b2000, 0x381, 0x1000)\n\t/home/ubuntu/go/src/github.com/cosmos/gaia/vendor/github.com/tendermint/tendermint/p2p/peer.go:334 +0xbd\ngithub.com/cosmos/gaia/vendor/github.com/tendermint/tendermint/p2p.(*MConnection).recvRoutine(0xc439a28870)\n\t/home/ubuntu/go/src/github.com/cosmos/gaia/vendor/github.com/tendermint/tendermint/p2p/connection.go:475 +0x4a3\ncreated by github.com/cosmos/gaia/vendor/github.com/tendermint/tendermint/p2p.(*MConnection).OnStart\n\t/home/ubuntu/go/src/github.com/cosmos/gaia/vendor/github.com/tendermint/tendermint/p2p/connection.go:170 +0x187\n"

[melekes]

E[10-19|04:45:16.844] Failed to dial peer                          module=p2p address=104.131.86.166:46656 err="Error creating peer: dial tcp 104.131.86.166:46656: i/o timeo
ut"

[melekes]

Error creating peer: dial tcp 104.131.86.166:46656: i/o timeot

we could probably increase dial timeout from 3s to say 10s

[melekes]

Not sure what to do about the blockchain reactor timeout. Anyway, timeout in blockchain reactor should not halt the syncing.

[srmo]

Interesting, we hit this a few days ago but weren't sure if this may be influenced by our environment. We have 7 Validators, each validator uses all 6 other nodes as seed and one crashed. Trying to bring it up again always lead to a halt of the fastsync after a few minutes without any noticable error message.
Oddly enough, when reducing the seed configuration from 6 to only 1, the node synced successfully.
Do you think this might be relatd or should we rather open a new issue?

[ebuchman]

@srmo can you see if this is fixed for you on develop, and if not open a new issue? thanks

[vanipolnedi]

while i am connecting to other node i'm getting error like :
E[10-10|10:58:49.009] Dialing failed module=p2p addr=8a030208cf82bd3dd7c178d05d6c96383a1d7e63@10.40.20.109:26656 err="Error{dial tcp 10.40.20.109:26656: connect: no route to host}" attempts=7
any suggestions?? thanks in advance

[ebuchman]

connect: no route to host

Sounds like some kind of connection issue? Looks like you're using an internal IP address. Check your network settings/configuration/connectivity.

[hs-harsh]

I'm getting this error when i'm increasing validator node to 6. I tried increasing dial timeout to 10 sec. but that also didn't worked

node1 | E[2019-06-17|05:49:22.334] Dialing failed module=pex addr=5b06f05a20529241666d7e558cfe46c26b17dc81@192.167.10.2:26656 err="dial tcp 192.167.10.2:26656: connect: connection refused" attempts=0 node1 | E[2019-06-17|05:49:22.335] Dialing failed module=pex addr=42942d8111c7cd7f72a7f05b8ed0878c82924261@192.167.10.4:26656 err="dial tcp 192.167.10.4:26656: connect: connection refused" attempts=0 node1 | E[2019-06-17|05:49:22.335] Dialing failed module=pex addr=87a22fc1737b31fb81e6dde4496cdc8e189a5afe@192.167.10.5:26656 err="dial tcp 192.167.10.5:26656: connect: connection refused" attempts=0 node1 | E[2019-06-17|05:49:22.336] Dialing failed module=pex addr=4871309e71b40434f3f3bfdaf7dbe0a1d83449e8@192.167.10.3:26656 err="self ID<4871309e71b40434f3f3bfdaf7dbe0a1d83449e8>" attempts=0 node3 | I[2019-06-17|05:49:22.367] Version info module=main software=0.31.5 block=10 p2p=7 node3 | I[2019-06-17|05:49:22.370] Starting Node module=main impl=Node node3 | I[2019-06-17|05:49:22.372] Started node module=main nodeInfo="{ProtocolVersion:{P2P:7 Block:10 App:1} ID_:87a22fc1737b31fb81e6dde4496cdc8e189a5afe ListenAddr:tcp://0.0.0.0:26656 Network:chain-RNpPWQ Version:0.31.5 Channels:4020212223303800 Moniker:3973a2612bff Other:{TxIndex:on RPCAddress:tcp://0.0.0.0:26657}}" node3 | E[2019-06-17|05:49:22.383] Dialing failed module=pex addr=87a22fc1737b31fb81e6dde4496cdc8e189a5afe@192.167.10.5:26656 err="self ID<87a22fc1737b31fb81e6dde4496cdc8e189a5afe>" attempts=0 node3 | E[2019-06-17|05:49:22.383] Dialing failed module=pex addr=5b06f05a20529241666d7e558cfe46c26b17dc81@192.167.10.2:26656 err="dial tcp 192.167.10.2:26656: connect: connection refused" attempts=0 node3 | E[2019-06-17|05:49:22.384] Dialing failed module=pex addr=42942d8111c7cd7f72a7f05b8ed0878c82924261@192.167.10.4:26656 err="dial tcp 192.167.10.4:26656: connect: connection refused" attempts=0 node2 | I[2019-06-17|05:49:22.403] Version info module=main software=0.31.5 block=10 p2p=7 node2 | I[2019-06-17|05:49:22.410] Starting Node module=main impl=Node node2 | I[2019-06-17|05:49:22.412] Started node module=main nodeInfo="{ProtocolVersion:{P2P:7 Block:10 App:1} ID_:42942d8111c7cd7f72a7f05b8ed0878c82924261 ListenAddr:tcp://0.0.0.0:26656 Network:chain-RNpPWQ Version:0.31.5 Channels:4020212223303800 Moniker:3973a2612bff Other:{TxIndex:on RPCAddress:tcp://0.0.0.0:26657}}" node2 | E[2019-06-17|05:49:22.414] Dialing failed module=pex addr=5b06f05a20529241666d7e558cfe46c26b17dc81@192.167.10.2:26656 err="dial tcp 192.167.10.2:26656: connect: connection refused" attempts=0 node2 | E[2019-06-17|05:49:22.419] Dialing failed module=pex addr=42942d8111c7cd7f72a7f05b8ed0878c82924261@192.167.10.4:26656 err="self ID<42942d8111c7cd7f72a7f05b8ed0878c82924261>" attempts=0 node0 | I[2019-06-17|05:49:22.471] Version info module=main software=0.31.5 block=10 p2p=7 node0 | I[2019-06-17|05:49:22.478] Starting Node module=main impl=Node node0 | I[2019-06-17|05:49:22.481] Started node module=main nodeInfo="{ProtocolVersion:{P2P:7 Block:10 App:1} ID_:5b06f05a20529241666d7e558cfe46c26b17dc81 ListenAddr:tcp://0.0.0.0:26656 Network:chain-RNpPWQ Version:0.31.5 Channels:4020212223303800 Moniker:3973a2612bff Other:{TxIndex:on RPCAddress:tcp://0.0.0.0:26657}}" node0 | E[2019-06-17|05:49:22.486] Dialing failed module=pex addr=5b06f05a20529241666d7e558cfe46c26b17dc81@192.167.10.2:26656 err="self ID<5b06f05a20529241666d7e558cfe46c26b17dc81>" attempts=0 node2 | E[2019-06-17|05:49:22.623] Error dialing peer module=p2p err="self ID<42942d8111c7cd7f72a7f05b8ed0878c82924261>" node1 | E[2019-06-17|05:49:23.184] Error dialing peer module=p2p err="self ID<4871309e71b40434f3f3bfdaf7dbe0a1d83449e8>" node1 | E[2019-06-17|05:49:23.335] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=0 node1 | E[2019-06-17|05:49:23.335] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=0 node3 | E[2019-06-17|05:49:23.384] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=0 node3 | E[2019-06-17|05:49:23.384] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=0 node2 | E[2019-06-17|05:49:23.413] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=0 node2 | E[2019-06-17|05:49:23.414] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=0 node0 | E[2019-06-17|05:49:23.483] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=0 node0 | E[2019-06-17|05:49:23.482] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=0 node0 | E[2019-06-17|05:49:23.583] Error dialing peer module=p2p err="self ID<5b06f05a20529241666d7e558cfe46c26b17dc81>" node3 | E[2019-06-17|05:49:24.312] Error dialing peer module=p2p err="self ID<87a22fc1737b31fb81e6dde4496cdc8e189a5afe>" node1 | E[2019-06-17|05:49:25.334] Error dialing peer module=p2p err="dial tcp 192.167.10.6:26656: connect: no route to host" node3 | E[2019-06-17|05:49:25.382] Error dialing peer module=p2p err="dial tcp 192.167.10.7:26656: connect: no route to host" node3 | E[2019-06-17|05:49:25.383] Error dialing peer module=p2p err="dial tcp 192.167.10.6:26656: connect: no route to host" node0 | E[2019-06-17|05:49:25.482] Error dialing peer module=p2p err="dial tcp 192.167.10.7:26656: connect: no route to host" node1 | E[2019-06-17|05:49:53.337] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=1 node1 | E[2019-06-17|05:49:53.337] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=1 node2 | E[2019-06-17|05:49:53.414] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=1 node2 | E[2019-06-17|05:49:53.414] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=1 node0 | E[2019-06-17|05:49:53.482] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=1 node0 | E[2019-06-17|05:49:53.483] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=1 node3 | E[2019-06-17|05:49:53.774] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=1 node3 | E[2019-06-17|05:49:53.775] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=1 node1 | E[2019-06-17|05:50:23.337] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=2 node1 | E[2019-06-17|05:50:23.337] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=2 node3 | E[2019-06-17|05:50:23.380] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=2 node3 | E[2019-06-17|05:50:23.380] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=2 node2 | E[2019-06-17|05:50:23.414] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=2 node2 | E[2019-06-17|05:50:23.414] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=2 node0 | E[2019-06-17|05:50:23.461] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: connect: no route to host" attempts=2 node0 | E[2019-06-17|05:50:23.483] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=2 node1 | E[2019-06-17|05:50:53.337] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=3 node1 | E[2019-06-17|05:50:53.337] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=3 node3 | E[2019-06-17|05:50:53.380] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=3 node3 | E[2019-06-17|05:50:53.380] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=3 node2 | E[2019-06-17|05:50:53.413] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=3 node2 | E[2019-06-17|05:50:53.413] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=3 node0 | E[2019-06-17|05:50:53.483] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=3 node3 | E[2019-06-17|05:51:22.457] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: connect: no route to host" attempts=4 node1 | E[2019-06-17|05:51:23.337] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=4 node1 | E[2019-06-17|05:51:23.339] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=4 node3 | E[2019-06-17|05:51:23.379] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=4 node2 | E[2019-06-17|05:51:23.413] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=4 node2 | E[2019-06-17|05:51:23.413] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=4 node0 | E[2019-06-17|05:51:23.485] Dialing failed module=pex addr=771b2782b4d7dec58ad8d9ced63bbddc3c6177d5@192.167.10.6:26656 err="dial tcp 192.167.10.6:26656: i/o timeout" attempts=4 node0 | E[2019-06-17|05:51:23.485] Dialing failed module=pex addr=74fb420b969bc0803cc82cfe279054a29553822a@192.167.10.7:26656 err="dial tcp 192.167.10.7:26656: i/o timeout" attempts=3

[abelakponine]

@melekes @vanipolnedi I know this is old but just incase someone needs this.

==== Fix new peer node connection already synced or Dial Error ====

I was having this same issue when adding new peers but was able to solve it with the following steps.

I realized I was changing the genesis.json file immediately after setting up tendermint which results in this issue on first run.

1. Run tendermint node after setup without changing genesis.json file i.e use the default genesis.json file,
2. you can run: rm -rf ~/.tendermint to delete the old setup, then run: tendermint init to generate a new genesis file
3. Run tendermint node to start the server. This will initialze tendermint with default and genesis.json file.
4. After initial run, stop server
5. Edit config.toml file of new peer node to your taste
6. Now copy the genesis of the main node into genesis.json file of the new peer node and save
7. Run tendermint unsafe_reset_all on new peer node
8. Run tendermint node on new peer node

This worked for me.

For Dialing Error

If you are still having a failed to dail error ...pex etc, this will be because of firewall blocking connection on your machine, ensure to add port 26656 through firewall in your machine and also run ufw allow 26656 on ssh