Skip to content

Statistics API

cryptocode edited this page Apr 16, 2018 · 12 revisions

Overview

The internal statistics API (rai::stat) provides a way for the C++ node to record and report detailed counter and sampling information. Statistics are optionally logged to separate stat files.

The "stats" RPC command can be used by external processes to query counters and samples. Useful for diagnostics, and monitoring/display in admin consoles.

API

  • A simple API is used to record events, such as bytes arriving at the network or blocks being put into the ledger. Events can be recorded at either the type level or the detail level. If recorded at the detail level, the count for the type level is updated as well.
  • Counters can be queried at the type or detail level.
  • Observers can register interest at the type or detail level, and for both counters and samples.
  • If configured, sampling is done at the configured interval automatically. N samples are kept in a ring buffer. There is no overhead if sampling is disabled.
  • A log sink interface allows counters and samples to be exported in various formats, currently CVS files and JSON objects.
  • An RPC action is available to report statistics as JSON

Examples

// Add to outbound traffic counter
this->node.stats.add (rai::stat::type::traffic, rai::stat::dir::out, size_a);

// Increment outbound publish message counter
node_l->stats.inc (rai::stat::type::message, rai::stat::detail::publish, rai::stat::dir::out);

// Sometimes details are updated only conditionally, so update the type-level first
node.stats.inc (rai::stat::type::error);
if (insufficient_work)
{
    node.stats.inc_detail_only (rai::stat::type::error, rai::stat::detail::insufficient_work);
}

// Querying a count in a test
ASSERT_EQ (0, system.nodes[0]->stats.count (rai::stat::type::error, rai::stat::detail::bad_sender));

// Observe inbound traffic samples. This will be called
// at the end of each interval.
node.stats.observe_sample (rai::stat::type::traffic, rai::stat::dir::in, [this_w](boost::circular_buffer<rai::stat_datapoint> & buff)
{
    // Use the array of samples, e.g. plot kbps
}

Configuration

"node": {
    ...
    "statistics": {
        "sampling": {
            "enabled": "true",
            "capacity": "5",
            "interval": "1000"
        },
        "log": {
            "headers": "true",
            "interval_counters": "5000",
            "interval_samples": "5000",
            "rotation_count": "5",
            "filename_counters": "counters.stat",
            "filename_samples": "samples.stat"
        }
    }
}

Future improvements / ideas

  • Make sampling rates, units, etc, configurable at both type and detail levels. Currently all event have the same interval.
  • Qt visualization of stats in the developer wallet. There's a prototype for traffic stats, but it requires a newer Qt version and I'd like the internal stats API to settle before completing this.
  • Pluggable log sinks. Currently files and json objects sinks are implemented. In the future, HTTP/TCP endpoint, etc can be added.
  • Enhance the RPC action to allow querying specific counters and samples.
  • Split key into constituents and pass to observer, since it may be observing on type-level, but want detail/direction info as well
  • Make stat_log_sink#to_object type safe. Possibly a good place to employ static polymorphism (CRTP)
  • Reduce lock granularity. Stat recording is fast, especially when sampling is turned off (default), but if contention becomes an issue, locking can be performed at the key level (type, detail, direction).
  • Websocket server to expose realtime stats?

Log file example

counters.stat

As specified in the example config, sampling interval is 1 second, stats are logged every 5 seconds, and the file rotates after 5 log cycles.

counters,2018.03.29 01:45:36
01:44:56,bootstrap,all,out,1
01:45:36,bootstrap,initiate,out,2
counters,2018.03.29 01:45:41
01:45:41,traffic,all,in,456344
01:45:41,traffic,all,out,189520
01:45:41,message,all,in,1925
01:45:41,message,all,out,1289
01:45:38,message,keepalive,in,165
01:45:41,message,keepalive,out,1027
01:45:41,message,publish,in,34
01:45:38,message,confirm_req,in,164
01:45:41,message,confirm_req,out,262
01:45:41,message,confirm_ack,in,1562
01:45:36,bootstrap,all,out,2
01:45:41,bootstrap,initiate,out,3

samples.stat

As specified in the example config, logging is done every 5 seconds and the sampling capacity is 5 (how many samplings are kept)

samples,2018.03.29 01:45:36
01:45:36,bootstrap,initiate,out,2
samples,2018.03.29 01:45:41
01:45:37,traffic,all,in,322608
01:45:38,traffic,all,in,37064
01:45:39,traffic,all,in,38752
01:45:40,traffic,all,in,25632
01:45:38,traffic,all,out,185072
01:45:39,traffic,all,out,3072
01:45:41,traffic,all,out,920
01:45:37,message,all,in,1387
01:45:38,message,all,in,126
01:45:39,message,all,in,179
01:45:40,message,all,in,101
01:45:37,message,all,out,1254
01:45:38,message,all,out,10
01:45:39,message,all,out,16
01:45:41,message,all,out,6
01:45:38,message,keepalive,in,165
01:45:38,message,keepalive,out,1011
01:45:39,message,keepalive,out,12
01:45:41,message,keepalive,out,3
01:45:37,message,publish,in,19
01:45:38,message,publish,in,8
01:45:40,message,publish,in,3
01:45:41,message,publish,in,4
01:45:38,message,confirm_req,in,164
01:45:37,message,confirm_req,out,249
01:45:38,message,confirm_req,out,3
01:45:39,message,confirm_req,out,6
01:45:41,message,confirm_req,out,3
01:45:37,message,confirm_ack,in,1046
01:45:38,message,confirm_ack,in,141
01:45:39,message,confirm_ack,in,150
01:45:40,message,confirm_ack,in,100
01:45:36,bootstrap,all,out,2
01:45:36,bootstrap,initiate,out,2
01:45:41,bootstrap,initiate,out,1

RPC

Query:

{ "action": "stats", "type": "counters" | "samples" }

Counters response

{
    "type": "counters",
    "created": "2018.03.29 01:46:36",
    "entries": [
        {
            "time": "01:46:36",
            "type": "traffic",
            "detail": "all",
            "dir": "in",
            "value": "3122792"
        },
        {
            "time": "01:46:36",
            "type": "traffic",
            "detail": "all",
            "dir": "out",
            "value": "203184"
        },
        {
            "time": "01:46:36",
            "type": "message",
            "detail": "all",
            "dir": "in",
            "value": "12494"
        },
        {
            "time": "01:46:36",
            "type": "message",
            "detail": "all",
            "dir": "out",
            "value": "1380"
        },
        {
            "time": "01:46:36",
            "type": "message",
            "detail": "keepalive",
            "dir": "in",
            "value": "172"
        },
        {
            "time": "01:46:36",
            "type": "message",
            "detail": "keepalive",
            "dir": "out",
            "value": "1084"
        },
        {
            "time": "01:46:36",
            "type": "message",
            "detail": "publish",
            "dir": "in",
            "value": "406"
        },
        {
            "time": "01:46:33",
            "type": "message",
            "detail": "confirm_req",
            "dir": "in",
            "value": "165"
        },
        {
            "time": "01:46:33",
            "type": "message",
            "detail": "confirm_req",
            "dir": "out",
            "value": "296"
        },
        {
            "time": "01:46:36",
            "type": "message",
            "detail": "confirm_ack",
            "dir": "in",
            "value": "11751"
        },
        {
            "time": "01:46:36",
            "type": "bootstrap",
            "detail": "all",
            "dir": "out",
            "value": "7965"
        },
        {
            "time": "01:46:36",
            "type": "bootstrap",
            "detail": "initiate",
            "dir": "out",
            "value": "7965"
        }
    ]
}

Samples response

{
    "type": "samples",
    "created": "2018.03.29 01:47:08",
    "entries": [
        {
            "time": "01:47:04",
            "type": "traffic",
            "detail": "all",
            "dir": "in",
            "value": "59480"
        },
        {
            "time": "01:47:05",
            "type": "traffic",
            "detail": "all",
            "dir": "in",
            "value": "44496"
        },
        {
            "time": "01:47:06",
            "type": "traffic",
            "detail": "all",
            "dir": "in",
            "value": "44136"
        },
        {
            "time": "01:47:07",
            "type": "traffic",
            "detail": "all",
            "dir": "in",
            "value": "18784"
        },
        {
            "time": "01:47:08",
            "type": "traffic",
            "detail": "all",
            "dir": "in",
            "value": "22680"
        },
        {
            "time": "01:47:03",
            "type": "traffic",
            "detail": "all",
            "dir": "out",
            "value": "4128"
        },
        {
            "time": "01:47:04",
            "type": "traffic",
            "detail": "all",
            "dir": "out",
            "value": "2136"
        },
        {
            "time": "01:47:05",
            "type": "traffic",
            "detail": "all",
            "dir": "out",
            "value": "1368"
        },
        {
            "time": "01:47:06",
            "type": "traffic",
            "detail": "all",
            "dir": "out",
            "value": "3504"
        },
        {
            "time": "01:47:08",
            "type": "traffic",
            "detail": "all",
            "dir": "out",
            "value": "1368"
        },
        {
            "time": "01:47:04",
            "type": "message",
            "detail": "all",
            "dir": "in",
            "value": "235"
        },
        {
            "time": "01:47:05",
            "type": "message",
            "detail": "all",
            "dir": "in",
            "value": "176"
        },
        {
            "time": "01:47:06",
            "type": "message",
            "detail": "all",
            "dir": "in",
            "value": "182"
        },
        {
            "time": "01:47:07",
            "type": "message",
            "detail": "all",
            "dir": "in",
            "value": "78"
        },
        {
            "time": "01:47:08",
            "type": "message",
            "detail": "all",
            "dir": "in",
            "value": "92"
        },
        {
            "time": "01:47:03",
            "type": "message",
            "detail": "all",
            "dir": "out",
            "value": "31"
        },
        {
            "time": "01:47:04",
            "type": "message",
            "detail": "all",
            "dir": "out",
            "value": "17"
        },
        {
            "time": "01:47:05",
            "type": "message",
            "detail": "all",
            "dir": "out",
            "value": "10"
        },
        {
            "time": "01:47:06",
            "type": "message",
            "detail": "all",
            "dir": "out",
            "value": "29"
        },
        {
            "time": "01:47:08",
            "type": "message",
            "detail": "all",
            "dir": "out",
            "value": "9"
        },
        {
            "time": "01:47:03",
            "type": "message",
            "detail": "keepalive",
            "dir": "in",
            "value": "10"
        },
        {
            "time": "01:47:04",
            "type": "message",
            "detail": "keepalive",
            "dir": "in",
            "value": "8"
        },
        {
            "time": "01:47:05",
            "type": "message",
            "detail": "keepalive",
            "dir": "in",
            "value": "7"
        },
        {
            "time": "01:47:06",
            "type": "message",
            "detail": "keepalive",
            "dir": "in",
            "value": "12"
        },
        {
            "time": "01:47:08",
            "type": "message",
            "detail": "keepalive",
            "dir": "in",
            "value": "2"
        },
        {
            "time": "01:47:03",
            "type": "message",
            "detail": "keepalive",
            "dir": "out",
            "value": "28"
        },
        {
            "time": "01:47:04",
            "type": "message",
            "detail": "keepalive",
            "dir": "out",
            "value": "16"
        },
        {
            "time": "01:47:05",
            "type": "message",
            "detail": "keepalive",
            "dir": "out",
            "value": "10"
        },
        {
            "time": "01:47:06",
            "type": "message",
            "detail": "keepalive",
            "dir": "out",
            "value": "28"
        },
        {
            "time": "01:47:08",
            "type": "message",
            "detail": "keepalive",
            "dir": "out",
            "value": "9"
        },
        {
            "time": "01:47:02",
            "type": "message",
            "detail": "publish",
            "dir": "in",
            "value": "11"
        },
        {
            "time": "01:47:04",
            "type": "message",
            "detail": "publish",
            "dir": "in",
            "value": "15"
        },
        {
            "time": "01:47:05",
            "type": "message",
            "detail": "publish",
            "dir": "in",
            "value": "7"
        },
        {
            "time": "01:47:06",
            "type": "message",
            "detail": "publish",
            "dir": "in",
            "value": "3"
        },
        {
            "time": "01:47:08",
            "type": "message",
            "detail": "publish",
            "dir": "in",
            "value": "11"
        },
        {
            "time": "01:46:56",
            "type": "message",
            "detail": "confirm_req",
            "dir": "in",
            "value": "44"
        },
        {
            "time": "01:47:00",
            "type": "message",
            "detail": "confirm_req",
            "dir": "in",
            "value": "5"
        },
        {
            "time": "01:47:01",
            "type": "message",
            "detail": "confirm_req",
            "dir": "in",
            "value": "3"
        },
        {
            "time": "01:47:03",
            "type": "message",
            "detail": "confirm_req",
            "dir": "in",
            "value": "3"
        },
        {
            "time": "01:47:05",
            "type": "message",
            "detail": "confirm_req",
            "dir": "in",
            "value": "1"
        },
        {
            "time": "01:46:55",
            "type": "message",
            "detail": "confirm_req",
            "dir": "out",
            "value": "4"
        },
        {
            "time": "01:47:00",
            "type": "message",
            "detail": "confirm_req",
            "dir": "out",
            "value": "12"
        },
        {
            "time": "01:47:01",
            "type": "message",
            "detail": "confirm_req",
            "dir": "out",
            "value": "3"
        },
        {
            "time": "01:47:03",
            "type": "message",
            "detail": "confirm_req",
            "dir": "out",
            "value": "3"
        },
        {
            "time": "01:47:05",
            "type": "message",
            "detail": "confirm_req",
            "dir": "out",
            "value": "1"
        },
        {
            "time": "01:47:04",
            "type": "message",
            "detail": "confirm_ack",
            "dir": "in",
            "value": "202"
        },
        {
            "time": "01:47:05",
            "type": "message",
            "detail": "confirm_ack",
            "dir": "in",
            "value": "164"
        },
        {
            "time": "01:47:06",
            "type": "message",
            "detail": "confirm_ack",
            "dir": "in",
            "value": "163"
        },
        {
            "time": "01:47:07",
            "type": "message",
            "detail": "confirm_ack",
            "dir": "in",
            "value": "65"
        },
        {
            "time": "01:47:08",
            "type": "message",
            "detail": "confirm_ack",
            "dir": "in",
            "value": "86"
        },
        {
            "time": "01:47:04",
            "type": "bootstrap",
            "detail": "all",
            "dir": "out",
            "value": "78"
        },
        {
            "time": "01:47:05",
            "type": "bootstrap",
            "detail": "all",
            "dir": "out",
            "value": "72"
        },
        {
            "time": "01:47:06",
            "type": "bootstrap",
            "detail": "all",
            "dir": "out",
            "value": "72"
        },
        {
            "time": "01:47:07",
            "type": "bootstrap",
            "detail": "all",
            "dir": "out",
            "value": "14"
        },
        {
            "time": "01:47:08",
            "type": "bootstrap",
            "detail": "all",
            "dir": "out",
            "value": "138"
        },
        {
            "time": "01:47:04",
            "type": "bootstrap",
            "detail": "initiate",
            "dir": "out",
            "value": "78"
        },
        {
            "time": "01:47:05",
            "type": "bootstrap",
            "detail": "initiate",
            "dir": "out",
            "value": "72"
        },
        {
            "time": "01:47:06",
            "type": "bootstrap",
            "detail": "initiate",
            "dir": "out",
            "value": "72"
        },
        {
            "time": "01:47:07",
            "type": "bootstrap",
            "detail": "initiate",
            "dir": "out",
            "value": "14"
        },
        {
            "time": "01:47:08",
            "type": "bootstrap",
            "detail": "initiate",
            "dir": "out",
            "value": "138"
        }
    ]
}
You can’t perform that action at this time.