Full Sidechain Replay Memory Usage

[inertia186]

As a private node operator, I would like to build the sidechain with my own local steemd node (without a third party blocks.log file), so that I can verify the sidechain.

The problem is that, by default, it will currently take about 60 hours to rebuild the sidechain using nothing but steemd as a source for transactions. This will only get longer over time.

This is complicated by the fact that we run out of memory after 24 hours, if we attempt to leave the node running that long. I believe there's a problem setting both of the following to 100 milliseconds ...

steemsmartcontracts/libs/Streamer.js

Line 12 in 775f20a

constructor(nodeUrl, currentBlock, antiForkBufferMaxSize = 2, pollingTime = 100) {

.
.
.

steemsmartcontracts/libs/Streamer.js

Lines 99 to 101 in 775f20a

	this.poller = setTimeout(() => {
	this.stream(reject);
	}, this.pollingTime);

steemsmartcontracts/plugins/Streamer.js

Line 196 in 0341df4

blockPoller = setTimeout(() => getBlock(reject), 100);

The problem is, if libs/Streamer.js manages to get ahead of plugins/Streamer.js, it will take more and more memory until it runs out.

I agree with the intent of 775f20a to speed up streaming. But making the two equal introduces a complication.

One might think, there's no way they could get ahead of each other because they both wait 100 milliseconds. But if plugins/Streamer.js processing takes even 1 millisecond longer than libs/Streamer.js, libs/Streamer.js will start to retain more and more blocks in its queue, over time.

If we are only 10,000 blocks behind, that's no problem. But if we are 30,000,000 blocks behind, after 24 hours, the node will crash.

I propose that libs/Streamer.js polling time should always be twice as long as plugins/Streamer.js polling time.