Add mempool statistics collector

[jonasschnelli]

This PR adds a statistics collector class which aims to collect various types of statistics up to the configurable maximum memory target. At the moment, only mempool statistics will be collected.

Motivation

Adding more statistics and visualization to the GUI would leverage its usage. To do so, we need stats that are collected even when the visualization is not visible (example: the GUI network graph will only draw data when it's visible which is kinda unusable)

How it works

This PR adds a simple stats manager that polls stats over a repetitive CScheduler task.

The samples are not guaranteed to be const-time. Each sample contains a time delta to the last one (uint16_t).

API

• -statsenable default disabled
• -statsmaxmemorytarget 10MB default maximal memory target to use for statistics.
• RPC: getmempoolstats
  ==

[
  {
    "percision_interval": 2,
    "time_from": 1494252401,
    "samples": [
      [
        11, 
        1, 
        1008, 
        0
      ], ....
   }...
]

Features

-> CScheduler driven sample collecting (current interval 2000ms)
-> Relevant mempool data (size, memory requirement, minfee) gets written to an atomic cache (no additional locking required)
-> Multiple precision levels (currently three, 2s, 60s, 1800s)
-> Memory target that will calculate how many samples do fit in the given target
-> Sample do only have a 2byte time-delta to the last sample, allowing to save some memory
-> Flexible design, adding more data-points should be simple (bandwidth, utxo-stats, etc.).

[isle2983]

is this semicolon unintentional?

[isle2983]

s/memmory/memory/

[isle2983]

s/statictis/statistics/

[isle2983]

s/consuption/consumption/

[isle2983]

s/detla/delta/

[isle2983]

This 'defaultStats' instance is not used.

[isle2983]

Hi Jonas,

I understand the concern in your description for the addMempoolSample() stat
bookkeeping designed to be as lightweight as possible in the critical execution
path. However, I have a few (perhaps under-informed, neophyte) questions which
would help me understand the design considerations better:

1. the comment in rpc_stats.cpp hints that the overhead of the JSON string
   generation is best optimized to be this 'flat' encoding as opposed to some
   encoding like:

{ "fieldNames" : ["delta_in_secs", "tx_count", "dynamic_mem_usage", "min_fee_per_k"], "samples" : [[val1, val2, val3, val4], [val1, val2, val3, val4], ] }

Is the 'flat' encoding strictly needed? or is there some other concern with
outputting a slightly more convenient format than 'flat'?

2. It appears possible to set the maximum memory target very hight such that
   many, many samples are collected and the overhead of the computation
   mempoolGetValuesInRange() inside the lock might become onerus (assuming I am
   correctly understanding how the lock works and the implications of holding it
   too long). Have you considered taking a copy of 'mempoolStats' in a way that
   lets you return the lock earlier, and doing the dataset computation outside the
   lock? (Is that even currently possible under the current execution model?)

Cheers,

Isle

[jonasschnelli]

@isle2983 Welcome to github.
Thanks for your feedback and your nitpicks. I really appreciate this and i'll process them during the next hours.

For your questions/inputs:

1. My idea with the JSON flat output was to bypass the JSON encoding/decoding.[val1, val2, val3, val4], [val1, val2, val3, val4], should also work. I just though a single string would result in faster encoding and decoding performance. But your approach seems to be the better choice, although not sure if we want to use UniValue for encoding or just appending strings... maybe we should start with the first and use a more optimized encoding if the JSON overhead is a problem.

2. Yes. That's a good point. Copying the samples vector could result in a memory peak when using large amount of -maxmemorytarget.

[MarcoFalke]

Would it make sense to add a line break after each sample's values?

[MarcoFalke]

Or just add the names of the columns as another entry in the dict.

Otherwise I fail to see how this rpc call is useful.

[isle2983]

I have been playing around making my own changes off these commits (isle2983:getmempoolstats). Mostly to just to get some hands on with the code and try to get my C++ up to par.

But anyway, I made the rpc output of the samples full JSON:

{
  "enabled": true,
  "maxmemorytarget": 10485760,
  "currentusage": 1734416,
  "time_from": 1471573271,
  "time_to": 1471657376,
  "sampleCount": 27131,
  "sampleFieldNames": [
    "timeDelta", 
    "txCount", 
    "dynMemUsage", 
    "minFeePerK"
  ],
  "samples": [
    [
      0, 
      1, 
      1728, 
      0
    ], 
    [
      4, 
      11, 
      15232, 
      0
    ], 
    ...
    (snip)
    ]
}

The JSON 'pretty' print through bitcoin-cli is definitely unwieldy. However, the computational overhead in doing the wrangling doesn't seem so bad.

The 1.7MM of stat data is from collecting just overnight. With that data, I can pull it off the node, parse and convert the JSON into CSV with a python script and plot it in gnuplot in under a second.

$ time myJunk/plotTxCount.sh 

real    0m0.966s
user    0m0.460s
sys     0m0.128s

Not sure what the comparable is with the qt gui stuff branch that is running, but this doesn't seem too bad on the face of it.

Also, if getting this info from the node to the UI quickly is a concern, perhaps a more dense, binary-like format is worth considering i.e:

{"stats_blob":"8b292cf....."}

One could imagine it being more efficient than even the 'flat' format, depending on the sophistication.

[jonasschnelli]

Thanks @isle2983 for the testing, benchmarks and improvements.
I have switched to the proposed array format for the samples (rather then the flat structure). A more performant binary format (inside of the JSON format) would be a hack. More performance would probably be possible over ZMQ.. but its currently a push only channel.

I also though again about copy the samples hash before filtering it. I came to the conclusion that it's not worth generating a memory peak (by a copy of the whole samples vector) in order to allow a faster release of the LOCK. The filtering should be very fast because it only compares some uint32 and does construct a new vector with a from-/to-iterators (should also preform fast).

[jonasschnelli]

Needed rebase.

[jonasschnelli]

Rebased.

[MarcoFalke]

Assigning "would be nice to have" for 0.14 per meeting today.

[morcos]

Just saw @gmaxwell's comment on #8550 (which I completely agree with) and it reminded to look at that PR and this one. Sorry for not getting involved sooner, but I really like the idea. Unfortunately I can think of many many stats (dozens and dozens) that we might want to collect, both to potentially show to users in whiz-bangy gui's and also would be useful for developers and businesses trying to understand unusual behavior on the network.

If we envision that there might be 1 KB of different stats data, then maybe rather than just saving sample data points and trimming when they hit memory usage, we should be smart about saving it along different time frames. For instance we could have second, minute, hour, day sampling intervals and we could save 1000 points or more on each and still have quite reasonable memory usage, but they could be auto trimmed.. so if you wanted to look at data from hours ago, you couldn't look at it on the second time frame...

[jonasschnelli]

@morcos: thanks for the comment. Yes. I completely agree. I think this is a first start and the current design allows features like you mentioned.
I once started with interpolating values instead of just trimming the back, you could in theory just reduce the "density" of the sample and interpolate the in-between values (to a point where this could make sense).
But yes, adding more stats probably require individual limits and trim-behaviours.

[morcos]

@jonasschnelli Well I guess what I was thinking was that one general framework might fit all stats. You log it with whatever frequency you want. And it's stored in up to 4 different histories (by second, minute, hour, day) and each of those is trimmed to some limit (say 1000 or 2000 data points each). Is there any type of stat that such a general framework might not work well with?

[jonasschnelli]

@morcos: the original idea I was trying to follow was to not collect in a fixed frequency. A) To avoid locking a thread for just collecting stats samples. B) To not collect over and over the same value if it was unchanged.
Take the traffic report as an example. If you like to collect stats of all peers traffic segmented into all available p2p commands, then you would probably "loose" plenty of memory by storing samples with identical values.

I had the idea of recording samples in the most restrained way possible. Collect lock free and only if values have changes; collect the according timestamp.
If you want to retrieve data with a fixed frequency/step-size, interpolate.

But not sure if this is stupid.

[jonasschnelli]

Rebased (main split)

[luke-jr]

c0af366 has unresolved conflicts :(

[paveljanik]

@luke-jr All checks have passed ;-)

[luke-jr]

@paveljanik They're removed in the subsequent commit.

[paveljanik]

Yes (I noticed that), this is why we should take Travis' results as a help only.

[morcos]

@jonasschnelli whatever happened to this plan?

09:19:49 < jonasschnelli> I think you convinced me to do the 1000s 1000m 1000h 1000d approach.
09:19:59 < jonasschnelli> maybe the 1000 is configurable.
09:20:04 < morcos> doesn't matter to me how we do it... i think a delta version coudl be just as good
09:20:14 < morcos> and you could just be smart about trimming the delta list or something
09:20:28 < morcos> yes, 1000 should be configurable i thik... actually maybe isn't enough for a default
09:20:45 < jonasschnelli> also, I liked the configurability of the buffer in MB.
09:20:52 < jonasschnelli> That's what you probably care about.
09:20:56 < jonasschnelli> Not the 1000
09:20:59 < morcos> 1000 secs is just 16 minutes...  you would not want to have to only have 16 data points
09:21:12 < jonasschnelli> You would say, I reserve 300MB for stats.
09:21:37 < jonasschnelli> Right... just en 
09:21:45 < jonasschnelli> just as an example
09:22:14 < jonasschnelli> So,.. you convinced me for high frequency recent and low frequency long time horizon,...
09:22:21 < morcos> ok cool...   any approach that automatically keeps both recent fine-grained and long time horizon bigger step, is fine with me

[jtimon]

Concept ACK

[laanwj]

Needs rebase / reply to @morcos' review comment

[jonasschnelli]

Completely rewrote this PR.

Current features are
-> CScheduler driven sample collecting (current interval 2000ms)
-> Relevant mempool data (size, memory requirement, minfee) gets written to an atomic cache (no additional locking required)
-> Multiple precision levels (currently three, 2s, 60s, 1800s)
-> Memory target that will calculate how many samples do fit in the given target
-> Sample do only have a 2byte time-delta to the last sample, allowing to save some memory
-> Flexible design, adding more data-points should be simple (bandwidth, utxo-stats, etc.).

ping @morcos

[ryanofsky]

Started looking at this PR, left a few comments.

[ryanofsky]

This seems like it would belong in AppInitParameterInteraction more than AppInitSanityChecks

[ryanofsky]

Should s/percision/precision/ throughout PR. Also maybe this field should be called "sample_interval" to be consistent with "samples" field below.

[ryanofsky]

Seems like this could be switched to unique_ptr.

[ryanofsky]

Should replace with for (unsigned int interval : precisionIntervals)

[ryanofsky]

Could simplify to vSamplesPerPrecision.emplace_back()

[ryanofsky]

I think you could drop the loop and just return {std::begin(precisionIntervals), std::end(precisionIntervals)};

[ryanofsky]

Using uint16_t here probably doesn't save any space due to struct padding.

[jonasschnelli]

Addressed @ryanofsky points.

[paveljanik]

Some C++11 magic needed here instead?

[jonasschnelli]

I think we better fix this together with all other Register*RPCCommands calls.

[paveljanik]

supermicronit: space before 10MB

[paveljanik]

statictics -> statistics

[paveljanik]

fallbackMaxSamplesPerPercision -> fallbackMaxSamplesPerPrecision

[jonasschnelli]

Fixed @paveljanik points and nits.

[paveljanik]

Wshadow emits:

stats/rpc_stats.cpp:78:42: warning: declaration shadows a variable in the global namespace [-Wshadow]

[jonasschnelli]

Fixed.

[ryanofsky]

Few more comments

[ryanofsky]

In commit "Add mempool statistics collector"

It seems like it would be more user friendly and extensible if this were an object instead of an array, so the elements could be named. It's a little awkard to have to remember that the first element in the array is time delta, second element is transaction count, etc.

[ryanofsky]

Thread #8501 (comment)

Any thoughts on this? Named fields seems more natural than tuples in a JSON data structure.

[ryanofsky]

In commit "Add mempool statistics collector"

Most of the other settings seem to be in seconds instead of milliseconds. Seems like it would be good to switch this one to seconds for consistency, and to be able to get rid of the division by 1000 in CStatsMempool::addMempoolSamples.

[ryanofsky]

Thread #8501 (comment)

Would be nice to use either seconds or milliseconds internally in the stats classes instead of both.

[ryanofsky]

In commit "Add mempool statistics collector"

Seems like it would be more efficient to switch to a circular buffer, or use std::deque instead of deleting from the beginning of the vector.

[ryanofsky]

Thread #8501 (comment)

Still worth considering alternatives to erasing from the beginning of a vector, I think.

[ryanofsky]

In commit "Add mempool statistics collector"

I don't understand what this is doing. It seems unnecessary, and strange because it is comparing the counter against numbers of seconds without taking collectInterval into account. If you will keep this, I think it'd be good to add a comment with an example of interval and counter values that explains how this is supposed to work.

[jonasschnelli]

I have added this to avoid an overflow of intervalCounter. I though resetting it when we exceeded the highest possible interval seems sane but I agree that a size_t overflow will very likely happens never... would you recommend to remove it?

[ryanofsky]

would you recommend to remove it?

I'd say remove it unless there's a way to reset it that makes sense (I'm sure there is one but nothing immediately comes to mind).

[ryanofsky]

Thread #8501 (comment)

Should follow up by removing this or adding comments suggested above.

[ryanofsky]

In commit "Add mempool statistics collector"

All three of these vectors have the same length (number of precision levels). Seems like it would be cleaner to have one vector containing a struct with all the information that needs to be stored for a given precision level. It would also make the c++ data structure more consistent with the json data structure returned from rpc.

[jonasschnelli]

That would be possible though I think that the current approach makes the core more readable.

[ryanofsky]

Do you think having parallel lists is more readable in c++ and having one list is more readable in json? Also, why are the first two vectors labeled perPrecision, and the third one not? I guess I think something like the following would be less ambiguous and duplicative:

struct PrecisionLevel {
    std::vector<CStatsMempoolSample> samples;
    size_t max_num_samples;
    int64_t last_sample_time;
};
std::vector<PrecisionLevel> m_precision_levels;

Could also rename cs_mempool_stats to cs_precision_levels to make it clearer what the lock is protecting.

[ryanofsky]

In commit "Add mempool statistics collector"

Would be good to move declaration closer to where this is being set (at least as close as possible).

[luke-jr]

Rebased on my repo.

git checkout stats_rpc && git fetch git://github.com/luke-jr/bitcoin stats_rpc && git reset --hard FETCH_HEAD && git push ...

[jonasschnelli]

Rebased (rebased @luke-jr's version).

[ryanofsky]

Should expand comment, unclear what it's referring to.

[ryanofsky]

Thread #8501 (comment)

Any thoughts on this? Named fields seems more natural than tuples in a JSON data structure.

[ryanofsky]

Thread #8501 (comment)

Would be nice to use either seconds or milliseconds internally in the stats classes instead of both.

[ryanofsky]

Maybe use unique_ptr

[ryanofsky]

Why have this as a separate variable when it seems to just be true when maxStatsMemory != 0. Maybe eliminate this or else add comments documenting the intended interfaction between the two variables.

[ryanofsky]

Should this be private since it is called by parameterInteraction?

Either way, it seems like this needs a comment about when it is safe to be called. E.g. it seems like this will not always work properly if it is called after startCollecting because it won't schedule the callback. But maybe it will work to change the size if it was previously greater than 0?

[ryanofsky]

Nobody seems to be listening to this signal. Should add comment explaining who intended listeners are, if keeping this.

[ryanofsky]

Indent and extra braces don't seem to accomplishing anything here.

[ryanofsky]

Maybe acquire LOCK before GetTime for more accurate time.

[ryanofsky]

Thread #8501 (comment)

Still worth considering alternatives to erasing from the beginning of a vector, I think.

[ryanofsky]

Thread #8501 (comment)

Should follow up by removing this or adding comments suggested above.

[ryanofsky]

Should be * sizeof(precisionIntervals[0])? And maybe multiplication should precede division for less rounding error.

[ryanofsky]

Should this acquire cs_mempool_stats before changing these values?

[ryanofsky]

Comment is wrong, and type should probably be int64_t, as returned by GetTime.

[ryanofsky]

Do you think having parallel lists is more readable in c++ and having one list is more readable in json? Also, why are the first two vectors labeled perPrecision, and the third one not? I guess I think something like the following would be less ambiguous and duplicative:

struct PrecisionLevel {
    std::vector<CStatsMempoolSample> samples;
    size_t max_num_samples;
    int64_t last_sample_time;
};
std::vector<PrecisionLevel> m_precision_levels;

Could also rename cs_mempool_stats to cs_precision_levels to make it clearer what the lock is protecting.

[ryanofsky]

s/percision/precision