Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move cs_vSend into SocketSendData and resolve RecordBytesSent lock inconsistency #19673

Closed
wants to merge 3 commits into from
Closed

Move cs_vSend into SocketSendData and resolve RecordBytesSent lock inconsistency #19673

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
75e6280
cleanup: Clean PushMessage
troygiorshev Aug 5, 2020
92897d4
Add missing lock annotations
troygiorshev Aug 26, 2020
9d47d6d
Move cs_vSend lock into SocketSendData
troygiorshev Aug 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
47 changes: 23 additions & 24 deletions src/net.cpp
View file
Open in desktop
Expand Up @@ -747,8 +747,9 @@ void V1TransportSerializer::prepareForTransport(CSerializedNetMsg& msg, std::vec
CVectorWriter{SER_NETWORK, INIT_PROTO_VERSION, header, 0, hdr};
}

size_t CConnman::SocketSendData(CNode *pnode) const EXCLUSIVE_LOCKS_REQUIRED(pnode->cs_vSend)
size_t CConnman::SocketSendData(CNode *pnode) const
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

9d47d6d

nit, clang-format suggests:

Suggested change
size_t CConnman::SocketSendData(CNode *pnode) const
size_t CConnman::SocketSendData(CNode* pnode) const

{
LOCK(pnode->cs_vSend);
auto it = pnode->vSendMsg.begin();
size_t nSentSize = 0;

Expand Down Expand Up @@ -1445,11 +1446,8 @@ void CConnman::SocketHandler()
//
if (sendSet)
{
Comment on lines 1447 to 1448
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

9d47d6d, nit:

        if (sendSet) {

LOCK(pnode->cs_vSend);
size_t nBytes = SocketSendData(pnode);
if (nBytes) {
RecordBytesSent(nBytes);
}
const size_t bytes_sent{SocketSendData(pnode)};
if (bytes_sent) RecordBytesSent(bytes_sent);
}

InactivityCheck(pnode);
Expand Down Expand Up @@ -2813,35 +2811,36 @@ bool CConnman::NodeFullyConnected(const CNode* pnode)

void CConnman::PushMessage(CNode* pnode, CSerializedNetMsg&& msg)
{
size_t nMessageSize = msg.data.size();
LogPrint(BCLog::NET, "sending %s (%d bytes) peer=%d\n", SanitizeString(msg.m_type), nMessageSize, pnode->GetId());
const size_t message_size{msg.data.size()};
LogPrint(BCLog::NET, "sending %s (%d bytes) peer=%d\n", SanitizeString(msg.m_type), message_size, pnode->GetId());

// make sure we use the appropriate network transport format
std::vector<unsigned char> serializedHeader;
pnode->m_serializer->prepareForTransport(msg, serializedHeader);
size_t nTotalSize = nMessageSize + serializedHeader.size();
std::vector<unsigned char> serialized_header;
pnode->m_serializer->prepareForTransport(msg, serialized_header);
const size_t total_size{message_size + serialized_header.size()};

size_t nBytesSent = 0;
bool optimistic_send{false};
{
LOCK(pnode->cs_vSend);
bool optimisticSend(pnode->vSendMsg.empty());
optimistic_send = pnode->vSendMsg.empty();

//log total amount of bytes per message type
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you retouch the branch, you may as well touch every line in this function:

Suggested change
//log total amount of bytes per message type
// Log total amount of bytes per message type

pnode->mapSendBytesPerMsgCmd[msg.m_type] += nTotalSize;
pnode->nSendSize += nTotalSize;
pnode->mapSendBytesPerMsgCmd[msg.m_type] += total_size;
pnode->nSendSize += total_size;

if (pnode->nSendSize > nSendBufferMaxSize)
if (pnode->nSendSize > nSendBufferMaxSize) {
pnode->fPauseSend = true;
pnode->vSendMsg.push_back(std::move(serializedHeader));
if (nMessageSize)
}
pnode->vSendMsg.push_back(std::move(serialized_header));
if (message_size) {
pnode->vSendMsg.push_back(std::move(msg.data));

// If write queue empty, attempt "optimistic write"
if (optimisticSend == true)
nBytesSent = SocketSendData(pnode);
}
}
// If write queue was empty, attempt "optimistic write"
if (optimistic_send) {
const size_t bytes_sent{SocketSendData(pnode)};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this change needed? Previously we'd optimistically write only one message, now we might write more than one message when other messages have been pushed by other threads.

if (bytes_sent) RecordBytesSent(bytes_sent);
}
if (nBytesSent)
RecordBytesSent(nBytesSent);
}

bool CConnman::ForNode(NodeId id, std::function<bool(CNode* pnode)> func)
Expand Down
6 changes: 3 additions & 3 deletions src/net.h
View file
Open in desktop
Expand Up @@ -759,8 +759,8 @@ class CNode
// socket
std::atomic<ServiceFlags> nServices{NODE_NONE};
SOCKET hSocket GUARDED_BY(cs_hSocket);
size_t nSendSize{0}; // total size of all vSendMsg entries
size_t nSendOffset{0}; // offset inside the first vSendMsg already sent
size_t nSendSize GUARDED_BY(cs_vSend){0}; // total size of all vSendMsg entries
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

92897d4

nit, clang-format suggests:

Suggested change
size_t nSendSize GUARDED_BY(cs_vSend){0}; // total size of all vSendMsg entries
size_t nSendSize GUARDED_BY(cs_vSend){0}; // total size of all vSendMsg entries

size_t nSendOffset GUARDED_BY(cs_vSend){0}; // offset inside the first vSendMsg already sent
uint64_t nSendBytes GUARDED_BY(cs_vSend){0};
std::deque<std::vector<unsigned char>> vSendMsg GUARDED_BY(cs_vSend);
RecursiveMutex cs_vSend;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this could use a comment. Something like:

//* Guards the send buffer and statistics about data sent on this connection */

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GUARDED_BY annotation seems quite self-documented; especially, when the mutex and all of the data members protected by it are nicely grouped :)

Expand Down Expand Up @@ -817,7 +817,7 @@ class CNode
std::atomic_bool fPauseSend{false};

protected:
mapMsgCmdSize mapSendBytesPerMsgCmd;
mapMsgCmdSize mapSendBytesPerMsgCmd GUARDED_BY(cs_vSend);
troygiorshev marked this conversation as resolved.
Show resolved Hide resolved
mapMsgCmdSize mapRecvBytesPerMsgCmd GUARDED_BY(cs_vRecv);

public:
Expand Down