Memory-efficient syncing

[hasyimibhar]

Syncing rounds currently requires the node to hold the entire diff in memory (both for sending and receiving). If the diff happens to be really big, it can easily overwhelm the node and taking it down. The objective of this PR to bound the memory usage of ledger during state syncing by writing to disk when necessary, while at the same time not sacrificing too much performance.

Major changes:

• State diff dump is now preceeded with <no. of diffs: 8 bytes><diff id: 16 bytes>.... This is done to allow the receiver to get the diff ids first without having to deserialize all the nodes.

• Diff is dumped into a buffer that is bounded in memory, but unbounded on disk.

• Once the diff is chunked, it's stored in a hash map which spills to on-disk KV if it exceeds the cache size.

• Downloaded chunks are written to a buffer that is bounded in memory, but unbounded on disk.

• Once all chunks are downloaded, it is deserialized into diff nodes one by one by populateDiffs instead of having to preload all the nodes first.

[codecov-io]

Codecov Report

Merging #177 into master will increase coverage by 0.23%.
The diff coverage is 73.33%.

@@            Coverage Diff             @@
##           master     #177      +/-   ##
==========================================
+ Coverage   48.62%   48.86%   +0.23%     
==========================================
  Files          39       39              
  Lines        6577     6586       +9     
==========================================
+ Hits         3198     3218      +20     
+ Misses       3033     3028       -5     
+ Partials      346      340       -6

Impacted Files	Coverage Δ
protocol.go	28.98% <0%> (ø)	⬆️
avl/node.go	72.38% <100%> (+0.96%)	⬆️
avl/tree.go	48.22% <66.66%> (-0.15%)	⬇️
tx_json.go	2.2% <0%> (ø)	⬆️
ledger.go	39.76% <0%> (+0.57%)	⬆️
graph.go	81% <0%> (+0.59%)	⬆️
gossip.go	94.44% <0%> (+8.33%)	⬆️
index.go	35% <0%> (+10%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1a64f98...2f566cd. Read the comment docs.

[hasyimibhar]

From a single run:

• Badger with 500MB diff:

  • SSD: 9s (accounts.Commit took 700ms)
  • HDD: 51s (accounts.Commit took 7.2s)

• LevelDB with 500MB diff:

  • SSD: 9.9s (accounts.Commit took 2.2s)
  • HDD: 1m20s (accounts.Commit took 40s)

[hasyimibhar]

Badger DB implementation is removed from this PR. Merge #217 instead to get Badger db.

[rkeene]

Tested successfully