New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

log bytes recv/sent per command #6589

Merged
merged 1 commit into from Dec 7, 2015

Conversation

Projects
None yet
@jonasschnelli
Member

jonasschnelli commented Aug 25, 2015

Not sure if we want to have this is master.

This adds a sent/recv byte counter per command to each CNode.
It is a waste product of some statistics gathering during finding a strategy for throttling nodes without affecting the overall quality of the network.

I think throttling the block response to a node with a height < self.height - 288 (TBD) would be a good approach. Nodes in initial sync might need longer to catch up if they are connected to throttled nodes only. Initial sync is a one time process and is therefore (maybe) be not very time critical.

Example:

    "bytessent_per_cmd": {
      "addr": 30003,
      "block": 28981980,
      "getheaders": 9682,
      "headers": 82,
      "inv": 30843,
      "ping": 48,
      "pong": 40,
      "tx": 97400,
      "verack": 0,
      "version": 103
    },
    "bytesrecv_per_cmd": {
      "getaddr": 0,
      "getdata": 11935,
      "getheaders": 997,
      "headers": 1458027,
      "ping": 40,
      "pong": 40,
      "reject": 230,
      "verack": 0,
      "version": 102
    }
@@ -2171,7 +2174,7 @@ void CNode::AbortMessage() UNLOCK_FUNCTION(cs_vSend)
LogPrint("net", "(aborted)\n");
}
void CNode::EndMessage() UNLOCK_FUNCTION(cs_vSend)
void CNode::EndMessage(const char* pszCommand) UNLOCK_FUNCTION(cs_vSend)

This comment has been minimized.

@sipa

sipa Aug 25, 2015

Member

Any reason not to use const std::string& ?

@sipa

sipa Aug 25, 2015

Member

Any reason not to use const std::string& ?

This comment has been minimized.

@jonasschnelli

jonasschnelli Aug 26, 2015

Member

CNode::BeginMessage() also uses const char* (https://github.com/bitcoin/bitcoin/blob/master/src/net.h#L499). EndMessage() is always called over PushMessage(const char* pszCommand, const T1& a1, ...). Not sure if it would make sense to cast/transform to a std::string.

@jonasschnelli

jonasschnelli Aug 26, 2015

Member

CNode::BeginMessage() also uses const char* (https://github.com/bitcoin/bitcoin/blob/master/src/net.h#L499). EndMessage() is always called over PushMessage(const char* pszCommand, const T1& a1, ...). Not sure if it would make sense to cast/transform to a std::string.

This comment has been minimized.

@sipa

sipa Aug 26, 2015

Member

ok

@sipa

This comment has been minimized.

@laanwj

laanwj Dec 4, 2015

Member

Right, I agree with @sipa that I prefer std::string, but yes for that to have any effect whatsoever that'd require switching that entire call chain to std::string and that is out of the scope of this pull.

@laanwj

laanwj Dec 4, 2015

Member

Right, I agree with @sipa that I prefer std::string, but yes for that to have any effect whatsoever that'd require switching that entire call chain to std::string and that is out of the scope of this pull.

This comment has been minimized.

@paveljanik

paveljanik Dec 5, 2015

Contributor

On the other hand, we are already changing EndMessage() here, so it could be done in one step, together.

@paveljanik

paveljanik Dec 5, 2015

Contributor

On the other hand, we are already changing EndMessage() here, so it could be done in one step, together.

This comment has been minimized.

@jonasschnelli

jonasschnelli Dec 5, 2015

Member

I think it's to invasive for a little logging feature. It would basically require to change all PushMessage(...) then BeginMessage() and EndMessage() function. An either conversion of CMessageHeader (that it accepts std::string) or some conversions to c-strings in the mentioned functions.
IMO, it's not worth changing this.

@jonasschnelli

jonasschnelli Dec 5, 2015

Member

I think it's to invasive for a little logging feature. It would basically require to change all PushMessage(...) then BeginMessage() and EndMessage() function. An either conversion of CMessageHeader (that it accepts std::string) or some conversions to c-strings in the mentioned functions.
IMO, it's not worth changing this.

This comment has been minimized.

@paveljanik

paveljanik Dec 5, 2015

Contributor

Agreed.

@paveljanik

paveljanik Dec 5, 2015

Contributor

Agreed.

@sipa

View changes

Show outdated Hide outdated src/net.cpp
@sipa

View changes

Show outdated Hide outdated src/net.cpp
@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Aug 25, 2015

Member

Concept ACK, I think this is useful.

Member

sipa commented Aug 25, 2015

Concept ACK, I think this is useful.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Aug 26, 2015

Member

Thanks @sipa for reviewing!
Fixed nits. Added bytessent_per_cmd and bytesrecv_per_cmd to getpeersinfo help.

Member

jonasschnelli commented Aug 26, 2015

Thanks @sipa for reviewing!
Fixed nits. Added bytessent_per_cmd and bytesrecv_per_cmd to getpeersinfo help.

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Aug 26, 2015

Member
Member

sipa commented Aug 26, 2015

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Aug 26, 2015

Member

@sipa: good point! Will address it soon.

Member

jonasschnelli commented Aug 26, 2015

@sipa: good point! Will address it soon.

@theuni

This comment has been minimized.

Show comment
Hide comment
@theuni

theuni Aug 26, 2015

Member

These maps will need locking

Edit: I take that back. It looks like the cs_vNodes in CopyNodeStats is enough. Might want to make CNode::mapSendBytesPerCmd / CNode::mapRecvBytesPerCmd private though, so nobody's tempted to use them directly.

Member

theuni commented Aug 26, 2015

These maps will need locking

Edit: I take that back. It looks like the cs_vNodes in CopyNodeStats is enough. Might want to make CNode::mapSendBytesPerCmd / CNode::mapRecvBytesPerCmd private though, so nobody's tempted to use them directly.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Aug 27, 2015

Member

@theuni: Right. I was aware of the locking and did come to the conclusion that both new maps are protected by cs_vSend and cs_vRecvMsg.

Moved the maps to the private section.

Member

jonasschnelli commented Aug 27, 2015

@theuni: Right. I was aware of the locking and did come to the conclusion that both new maps are protected by cs_vSend and cs_vRecvMsg.

Moved the maps to the private section.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Aug 27, 2015

Member

Added memory DOS prevention by filtering valid commands.

Member

jonasschnelli commented Aug 27, 2015

Added memory DOS prevention by filtering valid commands.

Show outdated Hide outdated src/net.cpp
@jgarzik

This comment has been minimized.

Show comment
Hide comment
@jgarzik

jgarzik Oct 1, 2015

Contributor

ut ACK - this actually recreates a change I wrote a long time ago (first version of getpeerinfo did this)

Contributor

jgarzik commented Oct 1, 2015

ut ACK - this actually recreates a change I wrote a long time ago (first version of getpeerinfo did this)

@dcousens

This comment has been minimized.

Show comment
Hide comment
@dcousens

dcousens Oct 4, 2015

Contributor

concept ACK

edit: is it worth persisting this to storage?

Contributor

dcousens commented Oct 4, 2015

concept ACK

edit: is it worth persisting this to storage?

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Dec 3, 2015

Member

IMO
for (unsigned int i = 0; i < sizeof(logAllowIncommingCmds)/sizeof(logAllowIncommingCmds[0]); i++) is fine until we switch over to C++11.

@dcousens: not sure if disk persistence would make sense here. I'd like to extend this up to the GUI layer to have a better network/bandwidth graph that could also generate and show data when it's not opened, though, this PR does not include time based data logging.

Member

jonasschnelli commented Dec 3, 2015

IMO
for (unsigned int i = 0; i < sizeof(logAllowIncommingCmds)/sizeof(logAllowIncommingCmds[0]); i++) is fine until we switch over to C++11.

@dcousens: not sure if disk persistence would make sense here. I'd like to extend this up to the GUI layer to have a better network/bandwidth graph that could also generate and show data when it's not opened, though, this PR does not include time based data logging.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Dec 4, 2015

Member

Passed Travis now.
Would be nice if someone can test this.

Member

jonasschnelli commented Dec 4, 2015

Passed Travis now.
Would be nice if someone can test this.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Dec 4, 2015

Member

Going to test this

Member

laanwj commented Dec 4, 2015

Going to test this

@laanwj

View changes

Show outdated Hide outdated src/net.cpp
@laanwj

View changes

Show outdated Hide outdated src/net.h
@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Dec 4, 2015

Member

@laanwj: I like your map.find approach.
Implemented (squash-me commit) and renamed POSIX reserved *_t type.

Member

jonasschnelli commented Dec 4, 2015

@laanwj: I like your map.find approach.
Implemented (squash-me commit) and renamed POSIX reserved *_t type.

@laanwj

View changes

Show outdated Hide outdated src/rpcnet.cpp
@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Dec 4, 2015

Member

Minimally-tested ACK, I like this functionality a lot

Member

laanwj commented Dec 4, 2015

Minimally-tested ACK, I like this functionality a lot

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Dec 4, 2015

Member

Speaking of which, why is verack accounted for 0 bytes? I guess we don't include the message header in the size? I think it should.

    "bytessent_per_cmd": {
      "addr": 31,
      "getheaders": 997,
      "inv": 9043,
      "ping": 16,
      "verack": 0,
      "version": 115
    },
Member

laanwj commented Dec 4, 2015

Speaking of which, why is verack accounted for 0 bytes? I guess we don't include the message header in the size? I think it should.

    "bytessent_per_cmd": {
      "addr": 31,
      "getheaders": 997,
      "inv": 9043,
      "ping": 16,
      "verack": 0,
      "version": 115
    },
@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Dec 4, 2015

Member

Correct. Right now only the message size counts... so your right. This needs to be fixed, otherwise we are not really counting net bytes. Working on it.

Member

jonasschnelli commented Dec 4, 2015

Correct. Right now only the message size counts... so your right. This needs to be fixed, otherwise we are not really counting net bytes. Working on it.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Dec 4, 2015

Member

It's a fixed number per packet, so should be straightforward enough :)

Member

laanwj commented Dec 4, 2015

It's a fixed number per packet, so should be straightforward enough :)

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Dec 4, 2015

Member

Added the header size to the bytes addition. Squashed to one commit.

Member

jonasschnelli commented Dec 4, 2015

Added the header size to the bytes addition. Squashed to one commit.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Dec 4, 2015

Member

Implemented @laanwj idea. Added a category "*unknown*" to aggregate bytes of received command that are not known. This could help identifying misbehaving nodes.

Member

jonasschnelli commented Dec 4, 2015

Implemented @laanwj idea. Added a category "*unknown*" to aggregate bytes of received command that are not known. This could help identifying misbehaving nodes.

@laanwj

View changes

Show outdated Hide outdated src/net.cpp
@paveljanik

This comment has been minimized.

Show comment
Hide comment
@paveljanik

paveljanik Dec 5, 2015

Contributor

@jonasschnelli To be consistent, please also change recvPerCmd.

Contributor

paveljanik commented Dec 5, 2015

@jonasschnelli To be consistent, please also change recvPerCmd.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Dec 5, 2015

Member

@paveljanik: Thanks of the review! Change and force pushed the last commit.

Member

jonasschnelli commented Dec 5, 2015

@paveljanik: Thanks of the review! Change and force pushed the last commit.

@paveljanik

This comment has been minimized.

Show comment
Hide comment
@paveljanik

paveljanik Dec 5, 2015

Contributor

ACK
This is very useful, thanks for it!

Contributor

paveljanik commented Dec 5, 2015

ACK
This is very useful, thanks for it!

@MarcoFalke

View changes

Show outdated Hide outdated src/rpcnet.cpp
@GIJensen

This comment has been minimized.

Show comment
Hide comment
@GIJensen

GIJensen commented Dec 5, 2015

ACK

@GIJensen

View changes

Show outdated Hide outdated src/rpcnet.cpp
@paveljanik

View changes

Show outdated Hide outdated src/net.h
@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Dec 7, 2015

Member

Fixed minor code style issues (three unnecessary spaces).

Member

jonasschnelli commented Dec 7, 2015

Fixed minor code style issues (three unnecessary spaces).

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Dec 7, 2015

Member

Will merge after squash and travis OK

Member

laanwj commented Dec 7, 2015

Will merge after squash and travis OK

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Dec 7, 2015

Member

Squashed.

Member

jonasschnelli commented Dec 7, 2015

Squashed.

@laanwj laanwj merged commit ca188c6 into bitcoin:master Dec 7, 2015

1 check was pending

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details

laanwj added a commit that referenced this pull request Dec 7, 2015

Merge pull request #6589
ca188c6 log bytes recv/sent per command (Jonas Schnelli)

laanwj added a commit to laanwj/bitcoin that referenced this pull request Dec 7, 2015

net: Account for `sendheaders` `verack` messages
Looks like these were forgotten in #6589.

@laanwj laanwj referenced this pull request Dec 18, 2015

Closed

Banning other user agents #7227

GIJensen added a commit to GIJensen/bitcoin that referenced this pull request Dec 23, 2015

net: Account for `sendheaders` `verack` messages
Looks like these were forgotten in #6589.
@@ -2480,6 +2503,9 @@ void CNode::EndMessage() UNLOCK_FUNCTION(cs_vSend)
unsigned int nSize = ssSend.size() - CMessageHeader::HEADER_SIZE;
WriteLE32((uint8_t*)&ssSend[CMessageHeader::MESSAGE_SIZE_OFFSET], nSize);
//log total amount of bytes per command
mapSendBytesPerMsgCmd[std::string(pszCommand)] += nSize + CMessageHeader::HEADER_SIZE;

This comment has been minimized.

@rebroad

rebroad Oct 16, 2016

Contributor

why not put ssSend.size() into a variable and use that?

@rebroad

rebroad Oct 16, 2016

Contributor

why not put ssSend.size() into a variable and use that?

@rebroad

This comment has been minimized.

Show comment
Hide comment
@rebroad

rebroad Oct 16, 2016

Contributor

Is anyone using this functionality? If so, can uses be documented somewhere?

Contributor

rebroad commented Oct 16, 2016

Is anyone using this functionality? If so, can uses be documented somewhere?

@MarcoFalke

This comment has been minimized.

Show comment
Hide comment
@MarcoFalke

MarcoFalke Oct 16, 2016

Member

@rebroad I think the goal was to find out what msg consumes the most bytes.

Member

MarcoFalke commented Oct 16, 2016

@rebroad I think the goal was to find out what msg consumes the most bytes.

kyuupichan referenced this pull request in kyuupichan/BitcoinUnlimited Mar 19, 2017

Merge pull request #6589
ca188c6 log bytes recv/sent per command (Jonas Schnelli)

kyuupichan referenced this pull request in kyuupichan/BitcoinUnlimited Mar 19, 2017

net: Account for `sendheaders` `verack` messages
Looks like these were forgotten in #6589.

OlegGirko pushed a commit to OlegGirko/dash that referenced this pull request Jun 26, 2017

net: Account for `sendheaders` `verack` messages
Looks like these were forgotten in #6589.

UdjinM6 added a commit to dashpay/dash that referenced this pull request Jun 29, 2017

Backport Bitcoin PRs #6589, #7180 and remaining part of #7181: enable…
… per-command byte counters in `CNode` (#1496)

* log bytes recv/sent per command

* net: Account for `sendheaders` `verack` messages

Looks like these were forgotten in #6589.

* Backport remaining part of Bitcoin PR bitcoin/bitcoin#7181.

Most of this PR is already merged, but a small part remaining
that makes per-command byte counts in CNode working.

Signed-off-by: Oleg Girko <ol@infoserver.lv>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment