Account history: option to prune old data (#292)

[abitmore]

PR for #292.

Changes:

• Added a max-ops-per-account option to account_history plugin, when set, the plugin will keep at most that number of entries per account. Set it to 0 to store no history. Set it to a big number to keep all histories.
  • if partial-operations option is set to true at the same time, old operation_history_object data (1.11.x) will be removed as well
• Added a removed_ops field to account_statistics_object to track removed number of entries, behavior of total_ops field didn't change.
• Added an index to account_transaction_history_multi_index_type container, to help tracking references to operation_history_object entries
• Adjusted get_relative_account_history node API to adapt the change
• Code refactoring
  • always call use_next_id() instead of creating then removing for (proposed) operations that didn't pass validation, to keep 1.11.x numbering consistent.
  • moved duplicate code to a new add_account_history function
• Minor bug fix:
  • if track-account and partial-operations are both set, now won't create an operation_history_object entry when a non-tracking new account is created.

Effects:

• with max-ops-per-account = 1000 and partial-operations = true, and without track-account set, a BitShares node now consumes around 3.5G of RAM. So it will be easier to provide common API service.

Limitations / Cons:

• after changed the value of the options, if the code is restarted without a replay, data will be inconsistent.
• when the options aren't set, the node consumes more memory than before due to additional index.

Review / test needed.

[oxarbitrage]

excellent work. anything to have options to reduce ram i think is good. i confirmed some of the statements made in the pull request of memory reductions but with some differences, not sure how are you measuring it, i am doing pmap [WITNESS PID]

• without options the node consumes a bit more of ram than before(less than 1 gig more).
• if you run it without options and then with options the new options will not take effect, need replay to switch between setups.
• by using max-ops-per-account alone makes a small difference decreasing memory usage from 18-19 gigs to 15-16 gigs. That setting alone does not have too much impact,so i tried as described with max-ops-per-account and partial-operations set to true as:

programs/witness_node/witness_node --data-dir data/my-blockprod --rpc-endpoint "127.0.0.1:8090" --enable-stale-production -w \""1.6.0"\" \""1.6.1"\" \""1.6.2"\" \""1.6.3"\" \""1.6.4"\" --max-ops-per-account 1000 partial-operations true

instead of the 3.5 gigs stated this setup is consuming around 16 gigs of ram(same as without , i was expecting a deeper decrease but actually this have sense to me, by having 1000 operations per account should be like an 80%-90% of the history).

a new test was made with:

programs/witness_node/witness_node --data-dir data/my-blockprod --rpc-endpoint "127.0.0.1:8090" --enable-stale-production -w \""1.6.0"\" \""1.6.1"\" \""1.6.2"\" \""1.6.3"\" \""1.6.4"\" --max-ops-per-account 100 partial-operations true

i was expecting significant lower memory here however after sync the witness node is consuming 16 gigs as well.

am i starting the node the wrong way ?

  total         16414672K
root@alfredo:~# pmap 26997

[abitmore]

@oxarbitrage your startup command is not correct, -- is missed before partial-operations.

[oxarbitrage]

silly error, sorry about that, testing again, thanks.

[oxarbitrage]

this is awesome, i am running the node with the last 100 ops per account as:

programs/witness_node/witness_node --data-dir data/my-blockprod --rpc-endpoint "127.0.0.1:8090" --enable-stale-production -w \""1.6.0"\" \""1.6.1"\" \""1.6.2"\" \""1.6.3"\" \""1.6.4"\" --max-ops-per-account 100 --partial-operations true

and the node is consuming 3.4 gigs. are you sure you got 3.5 with 1000 and not with 100 ?

i pointed the explorer code i have to the node with the changes and i have all the functionality for that ram, perfect for my needs that i am just pulling the last 20 operations of each account while developing.

thanks you very much for this, i think it can be merged.

[abitmore]

1000 or 100 doesn't matter much right now. When the option isn't enabled, most of memory is consumed by only a small number of accounts who have made hundreds of thousands or even millions of transactions:

• witnesses: price feeds
• trading bots: market orders

[oxarbitrage]

Yep i see now most of the history is market making/trading bots. El jun. 4, 2017 6:36 PM, "Abit" <notifications@github.com> escribió:

…

1000 or 100 doesn't matter much right now. When the option isn't enabled, most of memory is consumed by only a small number of accounts who have made hundreds of thousands or even millions of transactions: - witnesses: price feeds - trading bots: market orders — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#297 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AUrjaehCkjhlckRPr6-BpYno2KlQE5VEks5sAyNmgaJpZM4NujZr> .

[xeroc]

Wow! this is awesome progress!