Create multi-channel for built-in oracle nodes and consensus nodes communication

[Tommo-L]

Summary
Improve the communication of consensus nodes and oracle nodes.

Do you have any solution you want to propose?

As we discussed at NCA, we want to create separate channels for oracle nodes and high priority p2p message like consensus message through multi-channel technology.

Where in software does this update applies to?

• Consensus
• Built-in Oracle
• P2P (TCP)

Hi guys, let's discuss here @shargon @belane @igormcoelho @vncoelho @eryeer @doubiliu

[vncoelho]

Sounds a good idea, I think that @lightszero commented about TCP performance of Akka #611.
Maybe we could also open an issue on Akka repo to see what they suggest.

[shargon]

Agree, it could be beneficial in high stress scenarios. Taking into account that some tx must be 'high priority', if a cn is requesting it.

[shargon]

@Tommo-L I think that you can start working on this, and we can test the benefits with no-resilience

[Tommo-L]

yeah，we have already start working on it. @eryeer will send the report here after National Day.

[eryeer]

We have finished the Akka framework performance test, below is the test report:

• Performance of Actor in local machine

  • The number of messages transferred per second for single machine (from a local Actor to another local Actor): 3,860,000/s
  • The amount of data transferred per second for single machine (from a local Actor to another local Actor): 3.5GB/s

• Performace of single connection in TCP protocol

  • The number of messages transferred per second using a single connection: 80,000/s
  • The amount of data transfered per second using a single connection: 40MB/s

• Performance of multi-connection with single server port in TCP protocol

  • The number of messages transfered per second using two connections: 80,000-100,000/s for each connection, 160,000-200,000/s in total
  • The number of messages transfered per second using four connections: 80,000-100,000/s for each connection, 320,000-400,000/s in total

Note: The purpose of this test is to investigate whether TPS can be increased by increasing the number of TCP connections to a single server port. The test method is that the TCP server only opens one port, and the TCP client creates multiple Actors for TCP communication. Each Actor simultaneously sends messages to the binding port of the server, and checks the efficiency of the server receiving messages.

• Performance of multi-port in TCP protocol
  • The number of messages transfered per second using two ports: 80,000-900,000/s for each port, 160,000-180,000/s in total
  • The number of messages transfered per second using four ports: 80,000-900,000/s for each port, 320,000-360,000/s in total

Note: The purpose of this test is to investigate whether the TPS can be improved by increasing the opening of the server port. The method of investigation is that the TCP server opens multiple ports, and the TCP client creates a tcp actor for each server port (that is, creates a connection to each server port). Each TCP Actor simultaneously sends messages to the server port to which it is bound, and checks the efficiency of the server receiving the message.

• Machine for the performance test

  • CPU: Intel Core i5-8250 CPU @ 1.60GHz 1.80GHz
  • RAM: 8GB
  • Operation System: Windows10 X64

• Conclusion

  • The communication performance between local Actors of Akka messages is very good. According to Akka's official documentation, its performance is only limited by the hardware conditions of the machine on which the program runs. It can be said that it will not become a performance bottleneck for message delivery.
  • Akka TCP message transmission has a maximum number on a single connection, and the average number of messages per second per connection is 80,000-100,000/s.
  • Increasing the number of connections (that is, increasing the number of TCP Actors) can increase the speed of message transmission linearly. Under the connection of multiple connections, the maximum number of messages per second per connection does not decrease significantly.
  • Opening multiple ports actually creates multiple connections, so increasing the number of ports will also result in a linear increase in TPS.
  • In the further performance test, My CPU continued to be 100% occupied when testing more than 5 connections, which eventually led to the crash, so it failed to test the situation of more multiple connections.

[eryeer]

The above performance test is only for the performance of the Akka framework, the test code does not contain the business code of NEO. In next step we will specifically examine the Akka performance in NEO, check each of the various TCP actors' performance under high pressure, such as the message amount transfered per second per connection, the total amount of message traffic, etc., to confirm whether Akka TCP is a bottleneck.

[eryeer]

Now we finished the performance test of Akka TCP in Neo

Neo Akka Communication Performance Test

• Maximum amount of TCP message sent per second

  • Single Consensus Node Mode (plus one non-consensus node which is used as transaction sender in this test) : 1500/s (There is only one connection between the two nodes, so 1500/s is the maximum amount of communication for that connection.)
  • Four Consensus Nodes Mode : 900/s for each connection in each node

• Conclusion

  • Combined with the result of the previous performance test of Akka Framework, the maximum communication capability of Akka TCP for each connection is 80,000-100,000/s, then in the NEO program, under the highest TPS pressure, Maximum amount of TCP message of each channel for Four Consensus Nodes Mode is only 900/s. The load rate of the TCP connection is about 1.1%.
  • Even in the Single Consensus Node Mode, only one channel communicates between two nodes. The load of the TCP connection is still not very high. According to the analysis, some time-consuming logic in the code affects the processing speed (such as transaction verification, collection search and calculation, VM execution, etc.), which indirectly slows down the frequency of message communication between nodes.
  • So it seems Akka TCP is not the bottle neck at present. I think even though we adding oracle messages into Akka, it might not reach the maximun capability of Akka TCP.

@shargon @vncoelho What do you think? Any idea or opinion from all of you?

[vncoelho]

I think this nice empirical and experimental analyses deserves a paper or, at least, a medium post for now.

Let's keep Akka and all the good things it brought to us. In the future we may investigate more.

Congratulations to all NGD guys efforts!

[shargon]

Great work guys, you are able to add some profiler that it could help to find where this time was spent?

[eryeer]

@Qiao-Jin is working on the time spenting profiler, he can provide the analysis result and some optimization advice later.