P2P: parse network datastream into header/data components in socket thread #2016
Conversation
| vRecvMsg[vRecvMsg.size() - 1].complete()) | ||
| vRecvMsg.push_back(CNetMessage(SER_NETWORK, nRecvVersion)); | ||
|
|
||
| CNetMessage& msg = vRecvMsg[vRecvMsg.size() - 1]; |
…hread Replaces CNode::vRecv buffer with a vector of CNetMessage's. This simplifies ProcessMessages() and eliminates several redundant data copies. Overview: * socket thread now parses incoming message datastream into header/data components, as encapsulated by CNetMessage * socket thread adds each CNetMessage to a vector inside CNode * message thread (ProcessMessages) iterates through CNode's CNetMessage vector Message parsing is made more strict: * Socket is disconnected, if message larger than MAX_SIZE or if CMessageHeader deserialization fails (latter is impossible?). Previously, code would simply eat garbage data all day long. * Socket is disconnected, if we fail to find pchMessageStart. We do not search through garbage, to find pchMessageStart. Each message must begin precisely after the last message ends. ProcessMessages() always processes a complete message, and is more efficient: * buffer is always precisely sized, using CDataStream::resize(), rather than progressively sized in 64k chunks. More efficient for large messages like "block". * whole-buffer memory copy eliminated (vRecv -> vMsg) * other buffer-shifting memory copies eliminated (vRecv.insert, vRecv.erase)
|
@sipa: ITYM back(). Updated. |
| { | ||
| nRecvVersion = nVersionIn; | ||
| for (unsigned int i = 0; i < vRecvMsg.size(); i++) | ||
| vRecvMsg[i].SetVersion(nVersionIn); |
There was a problem hiding this comment.
I'm not sure this is right. When the receive version is set, it is only applied to new messages. In practice that probably doesn't mean anything, as the version/verack message order is quite strict.
|
I like the general design, but this pull mixes an client-side optimization with changing the network protocol policy. I'm not against making it more strict and not trying to resync after a partial message or garbage data, but maybe that needs some discussion at least. |
|
When I didn't have resync after garbage data in bitcoinj I did see failures due to it, though it was long ago. BTW, is it possible now to send huge numbers of messages and cause OOM conditions? Previously if you did that the unread data would stick around in the kernels socket buffers and be discarded automatically. Now I guess the socket thread can read faster than the main thread can process. |
|
The last time I saw garbage occurring frequently was after the feb20 protocol upgrade. I suppose we can start requiring no garbage now... The is still a receive buffer flooding check, by the way. |
|
Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/8af12a9f5a56de594700abb5b0d9a05adf82d64b for binaries and test log. |
|
I wonder why we even need that flood protection. You could just as well stop polling sockets for read events if their receive buffer is above some threshold. |
|
For the record: I saw a node segfault with (among others) this patch, in the ~CNetMessage destructor (the CDataStream in it was not allocated, I assume uninitialized memory used as a CNetMessage). |
| @@ -3380,7 +3366,10 @@ bool ProcessMessages(CNode* pfrom) | |||
| printf("ProcessMessage(%s, %u bytes) FAILED\n", strCommand.c_str(), nMessageSize); | |||
| } | |||
|
|
|||
| vRecv.Compact(); | |||
| // remove processed messages; one incomplete message may remain | |||
There was a problem hiding this comment.
Perhaps it's better to use an std::deque here instead of a std::vector?
|
Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/8af12a9f5a56de594700abb5b0d9a05adf82d64b for binaries and test log. |
|
Superceded by #2409 |
bitcoin#1962 changes the minimum required confirmations for InstantSend to 2 when running in testnet. This resulted in the quorum rank calculation to pick blocks which are higher then the current chain tip. This commit fixes this by calculating the added height based on nInstantSendConfirmationsRequired. This results in the value 4 for mainnet, so no incompatibility is introduced. On testnet and regtest, this results in 0.
…ete functional tests coverage 9c0c944 Tests: Add --all option to test_runner for complete coverage (random-zebra) Pull request description: Trivial, as per title. ``` ./test_runner.py --all ``` for the complete run of all the 62 (!) scripts in our suite. ACKs for top commit: Fuzzbawls: ACK 9c0c944 Tree-SHA512: 8b23c1923711d6c9081e27368f507e1231354e0f48a19370a7501b1a471c4be69e8c1fe7dcdd4a3a7a656b46009e581692e03169176fa60d3a5b9987f83c5f01
Replaces CNode::vRecv buffer with a vector of CNetMessage's. This simplifies
ProcessMessages() and eliminates several redundant data copies.
Overview:
header/data components, as encapsulated by CNetMessage
Message parsing is made more strict:
or if CMessageHeader deserialization fails (latter is impossible?).
Previously, code would simply eat garbage data all day long.
We do not search through garbage, to find pchMessageStart. Each
message must begin precisely after the last message ends.
ProcessMessages() always processes a complete message, and is more efficient:
rather than progressively sized in 64k chunks. More efficient
for large messages like "block".