need to be smarter about dialing (again)

[whyrusleeping]

We now have well over 200 nodes in the network. When adding large files (or lots of files) we frequently try to dial a lot of them, our dial limit is 10 addresses per peer, concurrently. Doing the math, thats 2000 sockets in the 'best' case, but since we have our own dial implementation, we have extra epoll fds sitting around. So its quite easy to hit that tiny little 1024 fd limit.

We should do a few things:

Dialing:

• lower the concurrent dial limit. to 5? (or maybe even 3)
• be smarter about dialing
  • localhost addresses should complete immediately, probably high priority
  • addresses on our own LAN should complete fairly quickly
  • addresses on other LANs should probably not be dialed, or at least prioritized lower
  • if we have a bunch of addresses that are the same ip with different ports, we should try and handle those differently.
• switch to udp-based! (udp will only need a single fd for all connections)
• disconnect from peers, no need to stay connected to every peer forever

[jbenet]

All of those +1.

I think the best solution is UDP.

TODO:

• make bindings for libutp.
• try quic bindings: https://github.com/devsisters/goquic

[jbenet]

• implement Avoid dialing known _local_ addresses when not in that network #1246
• implement Config option to disable dialing specific subnets entirely #1247

[cryptix]

Bump to stop dialing outselvs.

[BrendanBenshoof]

Another big thing you can do (if you are not already) is batching storage of blocks.

I designed a "batch store" algorithm a while back you could use here.

[jbenet]

@BrendanBenshoof this is unrelated to this problem. and should be addressed in a completely different layer

[whyrusleeping]

we have since implemented a much smarter dial limiter and scheduler, this isnt as much of an issue anymore