Adding disconnection reason and its use for ensuring better p2p performance

[Tommo-L]

Closes #1133

changes:

1. add disconnect reason
2. send disconnection message first, then close the connection
3. MaxConnectionReached and MaxPerAddressConnectionReached will attached with a random address list.

[erikzhang]

Why do you send this message?

[Tommo-L]

To create a connection an actor sends a Tcp.Connect message to the TCP Manager. Once the connection is established the connection actor sends a Tcp.Connected message to the commander, which registers the connection handler by replying with a Tcp.Register message. https://getakka.net/articles/networking/io.html#getting-started

We need to register the handler, otherwise we can't send the error message back.

[erikzhang]

Get it. But then it should be sent in LocalNode. Because the Peer doesn't send the message when disconnecting.

[Tommo-L]

No, we also need to send the register message in normal case#L233

private void OnTcpConnected(IPEndPoint remote, IPEndPoint local)
    ........
	ConnectedAddresses[remote.Address] = count + 1;
	IActorRef connection = Context.ActorOf(ProtocolProps(Sender, remote, local), $"connection_{Guid.NewGuid()}");
	Context.Watch(connection);
	Sender.Tell(new Tcp.Register(connection));  <------ here
	ConnectedPeers.TryAdd(connection, remote);
}

[erikzhang]

But we don't need to send Register when it is disconnecting.

[erikzhang]

The disconnection reason is sent in LocalNode.

[Tommo-L]

Yes, we send reason message in LocalNode, and create reason in Peer, as we have abstract the TcpDisconnect. I think it's okay to put Sender.Tell(new Tcp.Register(connection)); here, which is part of the akka handshake protocol.

[erikzhang]

It's illogical that register an actor and then close it immediately.

[Tommo-L]

We also want a better way, like HTTP various status codes, but akka doesn't support it, and we also want to carry the address list information.

[vncoelho]

@erikzhang, perhaps it is reaching the time to merge this PR for us to start testing it together with other daily routines.

[vncoelho]

@ShawnYun, with your knowledge in the Akka I believe you could give us some hand and ideas here.

[ShawnYun]

@vncoelho Thanks. But I don't know much about this either. So I ask the akka team, according to their answer, there are only two ways : sending the Tcp.Close message directly to close, or sending the Register and then send the reason via WriteCommand before sending Close.

[Tommo-L]

I'll split it into two prs, one is for duplicate connection, and this pr for disconnection reason.