Issues with 2.5.38 - Failed to pick a random peer from our list of 0 peers

[giovannirco]

after upgrading to 2.5.38 when starting with ark relay:run --token=ark --env=production --suffix=relay --network=mainnet It fails with the following logs

[2020-02-07 03:11:44.768] INFO : Starting Database Manager
[2020-02-07 03:11:44.771] INFO : Establishing Database Connection
[2020-02-07 03:11:56.971] INFO : Connecting to transaction pool
[2020-02-07 03:11:57.464] INFO : Starting P2P Interface
[2020-02-07 03:12:05.165] INFO : Socket worker started, PID: 180
[2020-02-07 03:12:05.661] INFO : Socket worker started, PID: 187
[2020-02-07 03:12:05.869] INFO : Starting Blockchain Manager :chains:
[2020-02-07 03:12:05.964] INFO : Verifying database integrity
[2020-02-07 03:13:29.682] INFO : Verified database integrity
[2020-02-07 03:13:29.694] INFO : Last block in database: 11,091,428
[2020-02-07 03:13:29.695] INFO : State Generation - Step 1 of 6: Block Rewards
[2020-02-07 03:14:05.087] INFO : State Generation - Step 2 of 6: Fees
[2020-02-07 03:14:36.272] INFO : State Generation - Step 3 of 6: Transfer
[2020-02-07 03:14:48.567] INFO : State Generation - Step 4 of 6: SecondSignature
[2020-02-07 03:14:49.684] INFO : State Generation - Step 5 of 6: DelegateRegistration
[2020-02-07 03:16:01.667] INFO : State Generation - Step 6 of 6: Vote
[2020-02-07 03:16:13.965] INFO : State Generation complete! Wallets in memory: 125341
[2020-02-07 03:16:13.966] INFO : Number of registered delegates: 1110
[2020-02-07 03:16:16.569] INFO : Transaction Pool Manager build wallets complete
[2020-02-07 03:16:16.650] INFO : Your network connectivity has been verified by 208.67.220.220
[2020-02-07 03:16:16.740] INFO : Your NTP connectivity has been verified by pool.ntp.org
[2020-02-07 03:16:16.741] INFO : Local clock is off by 3ms from NTP
[2020-02-07 03:16:19.463] INFO : Checking 0 peers
[2020-02-07 03:16:19.464] INFO : 0 of 0 peers on the network are responsive
[2020-02-07 03:16:19.464] INFO : Median Network Height: 0
[2020-02-07 03:16:21.970] INFO : Couldn't find enough peers. Falling back to seed peers.
[2020-02-07 03:16:22.975] ERROR: Could not download blocks: Failed to pick a random peer from our list of 0 peers: all are either banned or on a different chain than us
[2020-02-07 03:16:22.975] INFO : No new block found on this peer
[2020-02-07 03:16:22.977] INFO : Block download finished
[2020-02-07 03:16:22.980] INFO : Blockchain 100% in sync
[2020-02-07 03:16:23.761] INFO : Checking 0 peers
[2020-02-07 03:16:25.865] INFO : Public HTTP API Server running at: http://0.0.0.0:4003
[2020-02-07 03:16:25.865] INFO : Wallet API Server running at: http://0.0.0.0:4040
[2020-02-07 03:16:26.060] INFO : Webhooks are disabled
[2020-02-07 03:16:26.960] INFO : Exchange JSON-RPC Server is disabled
[2020-02-07 03:17:21.972] INFO : Checking 0 peers
[2020-02-07 03:17:23.068] INFO : Block download finished
[2020-02-07 03:17:23.267] INFO : Blockchain 100% in sync
[2020-02-07 03:17:24.972] INFO : Couldn't find enough peers. Falling back to seed peers.
[2020-02-07 03:18:23.272] INFO : Block download finished
[2020-02-07 03:18:23.275] INFO : Blockchain 100% in sync
[2020-02-07 03:18:24.973] INFO : Checking 0 peers
[2020-02-07 03:18:29.973] INFO : Couldn't find enough peers. Falling back to seed peers.
[2020-02-07 03:19:23.281] INFO : Block download finished
[2020-02-07 03:19:23.284] INFO : Blockchain 100% in sync

This is on dev cluster which I ran yarn install with this package.json:

{
  "dependencies": {
    "@arkecosystem/core": "2.5.38"
  }
}

On older versions with package.json referencing 2.5.31 this works flawlessly:

[2020-02-02 19:47:55.320] INFO : Starting Database Manager
[2020-02-02 19:47:55.324] INFO : Establishing Database Connection
[2020-02-02 19:48:06.715] INFO : Connecting to transaction pool
[2020-02-02 19:48:07.234] INFO : Starting P2P Interface
[2020-02-02 19:48:16.718] INFO : Socket worker started, PID: 81
[2020-02-02 19:48:17.018] INFO : Socket worker started, PID: 75
[2020-02-02 19:48:17.328] INFO : Starting Blockchain Manager :chains:
[2020-02-02 19:48:17.431] INFO : Verifying database integrity
[2020-02-02 19:48:29.450] INFO : Verified database integrity
[2020-02-02 19:48:29.456] INFO : Last block in database: 11,176,955
[2020-02-02 19:48:29.457] INFO : State Generation - Step 1 of 6: Block Rewards
[2020-02-02 19:48:34.034] INFO : State Generation - Step 2 of 6: Fees
[2020-02-02 19:49:00.115] INFO : State Generation - Step 3 of 6: Transfer
[2020-02-02 19:49:08.120] INFO : State Generation - Step 4 of 6: SecondSignature
[2020-02-02 19:49:09.054] INFO : State Generation - Step 5 of 6: DelegateRegistration
[2020-02-02 19:49:22.320] INFO : State Generation - Step 6 of 6: Vote
[2020-02-02 19:49:30.718] INFO : State Generation complete! Wallets in memory: 126075
[2020-02-02 19:49:30.719] INFO : Number of registered delegates: 1111
[2020-02-02 19:49:34.222] INFO : Transaction Pool Manager build wallets complete
[2020-02-02 19:49:34.898] INFO : Your network connectivity has been verified by 8.8.8.8
[2020-02-02 19:49:34.929] INFO : Your NTP connectivity has been verified by pool.ntp.org
[2020-02-02 19:49:34.930] INFO : Local clock is off by 3ms from NTP
[2020-02-02 19:49:42.325] INFO : Checking 153 peers
[2020-02-02 19:49:42.514] INFO : 153 of 153 peers on the network are responsive
[2020-02-02 19:49:42.515] INFO : Median Network Height: 11,176,972
[2020-02-02 19:49:42.516] INFO : Discovered 128 peers with v2.5.38.
[2020-02-02 19:49:42.516] INFO : Discovered 3 peers with v2.5.19.
[2020-02-02 19:49:42.517] INFO : Discovered 3 peers with v2.5.14.
[2020-02-02 19:49:42.517] INFO : Discovered 2 peers with v2.5.1.
[2020-02-02 19:49:42.518] INFO : Discovered 1 peer with v2.5.24.
[2020-02-02 19:49:42.518] INFO : Discovered 5 peers with v2.5.28.
[2020-02-02 19:49:42.519] INFO : Discovered 3 peers with v2.5.30.
[2020-02-02 19:49:42.519] INFO : Discovered 3 peers with v2.5.25.
[2020-02-02 19:49:42.519] INFO : Discovered 5 peers with v2.5.31.
[2020-02-02 19:49:44.924] INFO : Downloaded 18 new blocks accounting for a total of 1 transaction
[2020-02-02 19:49:45.020] INFO : Starting Round 219,157
[2020-02-02 19:49:46.520] INFO : Saving round 219,157
[2020-02-02 19:49:46.619] INFO : Block download finished
[2020-02-02 19:49:46.729] INFO : Blockchain 100% in sync
[2020-02-02 19:49:47.614] INFO : Checking 10 peers
[2020-02-02 19:49:50.523] INFO : Public HTTP API Server running at: http://0.0.0.0:4003
[2020-02-02 19:49:50.523] INFO : Wallet API Server running at: http://0.0.0.0:4040
[2020-02-02 19:49:50.715] INFO : Webhooks are disabled
[2020-02-02 19:49:51.224] INFO : Exchange JSON-RPC Server is disabled
[2020-02-02 19:50:46.816] INFO : Downloaded 8 new blocks accounting for a total of 0 transactions
[2020-02-02 19:50:46.824] INFO : Block download finished
[2020-02-02 19:50:46.924] INFO : Broadcasting block 11,176,981 to 153 peers
[2020-02-02 19:50:47.122] INFO : Blockchain 100% in sync
[2020-02-02 19:51:46.842] INFO : Downloaded 7 new blocks accounting for a total of 0 transactions
[2020-02-02 19:51:46.851] INFO : Block download finished
[2020-02-02 19:51:46.860] INFO : Blockchain 100% in sync
[2020-02-02 19:52:46.958] INFO : Downloaded 8 new blocks accounting for a total of 0 transactions

Since I couldnt make dev environment work I didnt upgrade my production node and we are running an older version

• Version used: 2.5.38
• Environment name and version (e.g. node.js v10.15.3): using docker image node:11-alpine
• Server type and version: Kubernetes v1.14.9 on EKS
• Operating System and version: using docker image node:11-alpine

[ghost]

Thanks for opening this issue! A maintainer will review this in the next few days and explicitly select labels so you know what's going on.

If no reviewer appears after a week, a reminder will be sent out.

[ksemaev]

I must say that currently only one of our 6 nodes faced this problem, others upgraded with no problems

[ksemaev]

Rollback with ark snapshot:rollback --number=10000 fixed the issue

[giovannirco]

TY Semaev, it is working fine now

[ghost]

This issue has been closed. If you wish to re-open it please provide additional information.