crawler not issuing disconnects

[ptschip]

Hello, I'm a developer on the Bitcoin Unlimited project. We've been looking at various types of connection slot attack mitigation and I noticed that the behavior of the bitnodes crawlers seems to be to NOT issue a disconnect when it's done making it's handshake and gathering whatever into it needs. This causes a slot to be used up unnecessarily but more importantly our slot mitigation software flags an that an eviction when another connection comes in to bump off the bitnodes connection. While this only happens when the slots are all full it would be great if the crawlers would disconnect on their own since we end up banning nodes (still in testing) for a short while if they exceed their eviction limits. If bitnodes crawlers would do their disconnect then we could set our limits lower and prevent more bad actors from trying to game the connection slots.

[ayeowch]

@ptschip Bitnodes uses up to 2 connection slots:

1. crawl.py - Connects to your node and disconnects after completing a handshake. This repeats after the full network snapshot is taken which happens approximately every 5 minutes given the current network size.

2. ping.py - Maintains open connection with your node to get latency data. The latency data is used to generate data for the leaderboard https://bitnodes.21.co/nodes/leaderboard/ and it is also presented in the individual node status page, e.g. https://bitnodes.21.co/nodes/62.63.157.78-8333/

From your description, it sounds like you are talking about ping.py? If so, this is currently an expected behavior from the crawler.

[ptschip]

ok, yeah i see it actually disconnecting after about 30 seconds...what was happening is that when we have no connection slots available the least useful connection gets evicted...so during that 30 second or so interval your connection gets evicted. I do wonder though why it's taking 30 seconds...i see the ping getting done but the connection still open for a while..are you sure you'r disconnecting right after the ping? Or maybe the ping received on your side from us is coming back much slower than the ping that we receive from you? Ok, that's probably what it is...but still seems like long time.

EDIT: I'm watching it a little more, sometimes it's about 15 seconds to get the disconnect, so it varies...I think yesterday it must have been a much slower day....Anyway, it looks like it's working the way you say it should. Sorry for the bother, just trying to understand.

[ptschip]

Taking a look through your code a bit, I'm wondering if the longer than expected connection times are due to the KeepAlive that you use and maybe you're not issuing an explicit disconnect and just allowing the connections to time out? What I see is that the ping, at least the one we receive from you, usually takes about 1 second but then it take 15 to 30 seconds for the connection to drop.

[ayeowch]

@ptschip Connection from ping.py will drop only when there is network issue. It is not expected to disconnect from your node, i.e. it takes up 1 connection slot indefinitely.