Fix the issue of high RAM usage and long runtime for huge bootstrap_balances.json file

[shiatcb]

Fix the issue of high RAM usage and long runtime for huge bootstrap_balances.json file.

Motivation

User reported 2 issues when running bootstrap_balances.json (101K accounts) for IOTA:

1. When memory_limit_disable=false, bootstrap process will stop at 35Kth accounts because of transaction storage limit.
2. After enabling memory_limit_disable, the running time of bootstrap process takes ~8h40m which is too long.

This ticket aims to resolve both issues so that:

1. Reduce the memory usage for bootstrap process to make it complete successfully.
2. Reduce the time of running bootstrap process for huge data set.

Links to the issues:
coinbase/mesh-cli#259
coinbase/mesh-cli#260

Investigation

In general, bootstrap balances consists of 3 steps:

1. Load and parse bootstrap_balances.json.
2. Store the records (account & currency & amount) in transaction object as a slice of entries.
3. Commit the transaction by writing all entries in LSM and vLog.

I've experimented in 32G RAM machine by bootstrapping different number of accounts and computed the duration of step 2) and 3). The result is as follows:

# acct            write entry            commit txn
10115             2m42s                  98.2ms
5059              33s                    48.4ms
2529              6s                     21ms
1264              1s                     9.5ms
632               340ms                  5.4ms
316               117ms                  2.2ms 

When running bootstrap scripts with 101K accounts, the process stopped at 35K records with %6.4 memory usage, which is ~2G.

Solution

Based on investigation, the solution is to split the huge number of accounts into different batches. Each batch contains 600 records. We commit each batch before reading the next batch. By reducing the number of accounts committed in each transaction, we can achieve a better memory usage (101K records used %4 of total RAM which is ~1.3G) and much better running time (~60s for 101K records).

[shrimalmadhur]

Closing and created to work on that #367