Experimental ledger pruning

[SergiySW]

This is the initial, experimental implementation of pruning, related to #1094. Node ledger pruning is designed to safely discard parts of the ledger. Now ledger size can be reduced from 23.8 GB to 7.5GB after vacuum. And from 18.8 GB to 5.2 GB after database rebuild.

Configuration and considerations

• This version of ledger pruning is for experimental use only:
• Enable optional ledger pruning with node CLI --debug_prune for manual pruning & node launch flag --enable_pruning for automatic node background actions.
• Pruning rules can be configured in [node.experimental] config with “max_pruning_age” (Time in seconds to limit for blocks age after pruning. Also period for automatic node pruning task, default 1 day, 5 minutes for beta network) & “max_pruning_depth” (Limit for full blocks in chain after pruning, default 0 - unlimited). By default all confirmed blocks older than 1 day can be pruned, except genesis block & last confirmed block in each account chain.
• Last confirmed block in account chain & all unconfirmed blocks cannot be pruned from ledger. Pruning is relying on confirmation height, so confirmation height reset or too low online_weight_minimum config can cause issues.
• Pruned blocks hashes are located in database table “pruned” & can be used by node internally ([Pruning] Basic pruning blocks storage & database upgrade #2946).

RPC, CLI and other changes

• Several RPC calls are modified to support pruned blocks related to main request block ([Pruning] Support pruned chains in RPC & QT wallet #2977).
• RPC “history”: if previous block is pruned, “amount” field is not returned for all blocks, “account” field is not returned for state blocks, also state block type/subtype is “unknown” . If source is pruned, then “account” field is not returned for receive/open legacy & for receive/open state blocks ([Pruning] Support pruned chains in RPC & QT wallet #2977).
• Same as above for QT wallet history ([Pruning] Support pruned chains in RPC & QT wallet #2977).
• RPC “block_info”/“blocks_info”: if previous block is pruned, “amount” field is not returned. RPC “block_count”: for nodes with enabled pruning returns 2 new fields: “full” blocks & “pruned” blocks. “full” + “pruned” = total “blocks” ([Pruning] Support pruned chains in RPC & QT wallet #2977).
• Block count for CLI/RPC/telemetry is sum of full blocks & pruned blocks ([Pruning] Support pruned chains in RPC & QT wallet #2977).
• RPC "pruned_exists" to chekc pruned block existence in local database.

Other notes

• Node is aggressively pruning ledger during initial bootstrap (until hardcoded block count) to limit node disk space usage, pruning config settings are ignored. If you want to avoid this behavior, finish full bootstrap before pruning start.
• --enable_pruning flag & config option “enable_voting” are mutually exclusive to prevent representatives start with pruned ledger. Pruning is not available for any representatives ([Pruning] Pruning configuration settings #2947).
• Pruned node can serve as bootstrap server for remaining blocks in ledger ([Pruning] Support pruned blocks in bootstrap actions #2976).
• Multiple internal functions are modified to use pruned blocks in block processor ([Pruning] Accept pruned-related blocks in blocks processing & rollback #2974), internal node wallet actions ([Pruning] Wallet receive action & search pending for pruned blocks #2992), confirmation height processor ([Pruning] Support pruned blocks in confirmation height processor & active transactions #2978) and rep crawler ([Pruning] Use pruned blocks in rep crawler #2975).
• Development: consider using ledger safe functions for code with possible pruned blocks ([Pruning] Ledger pruning functions #2968).

Additional updates being reviewed

• Include pruned count in telemetry.
• Improve RocksDB pruned block count calculation.