Troubleshoot/resolve "no space available" error during high message volume

[qwahzi]

Summary

Possibly related to TCP buffers (happens when writing to a TCP socket), still troubleshooting. Not a critical error, but needs improved error handling. When resolved, it should help node performance whenever volume is high (since it could reduce dropped traffic), but so far it's only reproducible in test environments (since testing has improved so much that we can push the node to its limits). Piotr has a working test branch that handles it better, but more testing is needed

May be related to the receiver not receiving fast enough (tried replicating with a cpu-bound test node). In theory the fixed error should just trigger a wait/retry (the socket is fine) vs the current socket close + reconnect. It may be result of using Boost ASIO - seems like an OS-level issue that requires custom error handling

The Nano nodes uses many very small packets, which may play a role in the above items. Dropping the number of TCP connections to the number of nodes doesn't seem to solve the error message, and reducing the number of votes per message causes errors to increase. Can't increase the votes per message currently, so it's difficult to test what would happen with bigger messages (e.g. 100 votes per message), but that's an area for future research