Issues with Peers dissapearing

[theblains]

Hi,

Been having an issue with having no peers after looking into it it seems it may be an issue with the duplicite IP protection. My log is fulled of the following when this occurs;

May 23 20:40:28 bitcoin-cash electrumx_server[12484]: ERROR:PeerManager:[bchx.disdev.org:50002 SSL] dupe peer: Peer bchx.disdev.org is a dupe! 1 other peers with IP 193.37.152.144 were found!
May 23 20:40:28 bitcoin-cash electrumx_server[12484]: ERROR:PeerManager:[blackie.c3-soft.com:50002 SSL] dupe peer: Peer blackie.c3-soft.com is a dupe! 1 other peers with IP 78.97.206.149 were found!
May 23 20:40:29 bitcoin-cash electrumx_server[12484]: ERROR:PeerManager:[electron.jochen-hoenicke.de:51002 SSL] dupe peer: Peer electron.jochen-hoenicke.de is a dupe! 1 other peers with IP 88.198.39.205 were found!
May 23 20:40:29 bitcoin-cash electrumx_server[12484]: ERROR:PeerManager:[electroncash.dk:50002 SSL] dupe peer: Peer electroncash.dk is a dupe! 1 other peers with IP 5.103.137.146 were found!
May 23 20:40:29 bitcoin-cash electrumx_server[12484]: ERROR:PeerManager:[electrumx-bch.cryptonermal.net:50002 SSL] dupe peer: Peer electrumx-bch.cryptonermal.net is a dupe! 1 other peers with IP 18.233.20.36 were found!
May 23 20:40:29 bitcoin-cash electrumx_server[12484]: ERROR:PeerManager:[electrumx.hillsideinternet.com:50002 SSL] dupe peer: Peer electrumx.hillsideinternet.com is a dupe! 1 other peers with IP 162.220.47.150 were found!
May 23 20:40:29 bitcoin-cash electrumx_server[12484]: ERROR:PeerManager:[bch0.kister.net:50002 SSL] dupe peer: Peer bch0.kister.net is a dupe! 1 other peers with IP 100.11.124.171 were found!
May 23 20:40:29 bitcoin-cash electrumx_server[12484]: ERROR:PeerManager:[bch.imaginary.cash:50002 SSL] dupe peer: Peer bch.imaginary.cash is a dupe! 1 other peers with IP 198.27.66.168 were found!

Not sure what is causing this to occur but once they are all flagged as dupes my peers go to 0. Any ideas what could be causing it to check already connected and existing peers and then making themselfs as duplicates?

Thanks

[imaginaryusername]

Interestingly I can see myself connected to you as I type from bch.imaginary.cash. Could you possibly catch when this is happening and see what they are duplicates of?

[cculianu]

I think it eventually "recovers" and the peers come back (which would explain why you two guys are connected now) -- but it does "explode" the log with "dupe peer" messages and leaves you nearly peerless for a little while when it happens

I know because I also got this error too once 2 days ago.

This is definitely some sort of bug. I will investigate it.

[cculianu]

@theblains -- did this happen frequently to you (I ask because I want to know how to reproduce it reliably).

I am going to make this a high priority today as it's a pretty confidence-reducing bug, even if it recovers (not sure if it does but I think the 1 time it happened to me it came back) ...

[theblains]

@imaginaryusername Ill try keep an eye on it unfortunately its hard to try catch it.

@cculianu It does happen quite frequently more so on the testnet i guess maybe due to less servers? Unfortunately i have looked through the logs but nothing jumps out at what may be causing it. If you have anything you would like me to try let me know.
I found a way to make it recover which is to send an add_peer command in RPC to a known good peer which will then pull all the servers from and re-peer you.

Thinking about the testnet wondering if its more common as once it blows up the peer list there is less servers that may have a peer list intact so it never recovers when on mainnet there is a higher chance of at least 1 server having an intact peer list which will then respread.

[cculianu]

@theblains ah ok, thanks for the follow-up. Can't think of anything obvious for you to try. I will audit the code -- hopefully the bug will jump out at me with any luck.

Just curious -- is my theory that it auto-recovers true? Did you restart each time or did it sometimes just recover on its own?

The thing is .. it doesn't ban these servers.. so it should auto-recover after a few minutes. I've left my server unattended for weeks without even checking it recently and I would imagine it happened and I never noticed because it recovers.

I want to know so I know how bad to feel about this bug.. :)

[theblains]

@cculianu Hi, I have been keeping a closer eye on whats going on. It does seem it recovers itself after a while on Main Net. The same cannot be said for testnet. I wonder if this is due to the less amount of peers who may have a correct peer list to re-seed everyone.

[cculianu]

:(

Yeah that's probably what's going on.. you're right.

I will fix this soon - a lot of other stuff is going on but it's on my list of important things to do.

[theblains]

@cculianu Greetings, Any update on this? Mainnet seems to be fine I guess the number of servers means it keeps re-seeding however testnet does not i generally have no peers on testnet.

Thanks

[cculianu]

Hi— @theblains I haven’t had time to work on this.

I’ll try and set up some time this week. It should hopefully be a quick bug fix.

[jkister]

for now we should whitelist peers in coins.py-- override blacklist.json and peer manager

[cculianu]

Sorry I haven't had time to give electronX love. It's humming along though.

I've been involved with designing/implementing its replacement... and there's a very smart volunteer programmer working on that now. Something that scales better. We'll see if that ends up being reliable enough to entirely replace ElectronX. It won't be written in python but rather run as native code (currently being written in Rust).

I'll see if I can add a fix for this to whitelist coins.py.. .esp. on testnet. And to turn off the other sybil defenses. Testnet is very unlikely to get abused anyway...

[cculianu]

This fork has been discontinued. See issue #20. I am recommending users either install Fulcrum or the latest ElectrumX. Closing thread...