Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Ledger Pruning #1094

Open
rkeene opened this issue Aug 23, 2018 · 12 comments
Open

Support Ledger Pruning #1094

rkeene opened this issue Aug 23, 2018 · 12 comments
Labels
enhancement
Milestone
Research for Futu...

Comments

@rkeene
Copy link
Contributor

@rkeene rkeene commented Aug 23, 2018

As a part of node stratification (#1092) not all nodes will need the full ledger, and a mechanism will need to be created to determine which parts of the ledger can be safely discarded on each node. This mechanism will be called pruning.

More details will be added as time goes on.
@rkeene rkeene added this to the V19.0 milestone Aug 23, 2018
@rkeene rkeene mentioned this issue Aug 23, 2018
Closed
@ghost
Copy link

@ghost ghost commented Aug 27, 2018

Are you also thinking of a mechanism to make pending deposits compatible with pruning?

Loading

@rkeene
Copy link
Contributor Author

@rkeene rkeene commented Aug 27, 2018

All mechanisms that involve pruning must not break the ledger. This includes not pruning the sending block for blocks that have no corresponding receiving block but which could in the future (i.e., we are free to locally prune sends to address 0, since there can be no receive block generated from that address).

Loading

@ghost
Copy link

@ghost ghost commented Aug 27, 2018

Has the core team ever discussed the possibility of making such that the receive block is automatically generated after its send block, by making changes in the protocol?

Loading

@cryptocode
Copy link
Member

@cryptocode cryptocode commented Aug 27, 2018

@Omarino99 account holder must sign the receive block, but may not be online/cold wallet/etc

Loading

@ghost
Copy link

@ghost ghost commented Aug 27, 2018

@cryptocode what I'm suggesting would be changing the protocol such that receive blocks don't require the signature of the account. I guess this has already been discussed? I know it would bring up different problems though.

Loading

@ghost
Copy link

@ghost ghost commented Aug 27, 2018 •
edited by ghost
Loading

@rkeene on the roadmap I find this:

Ledger Pruning allows for a much smaller database size. This is done through pruning off blocks other than the frontier (head block) +1 or 2 previous for each account. This feature is dependent on Universal Blocks as they create a snapshot of an account on a per block basis requiring fewer blocks to verify an account.

Isn't this just plain wrong? It would work only on certain accounts. Everyone who doesn't know the protocol believes that it would be possible to prune all accounts, while instead it's not possible.
Link: https://developers.nano.org/roadmap/ledger-pruning/

Loading

@rkeene
Copy link
Contributor Author

@rkeene rkeene commented Aug 27, 2018

It can be either considered incorrect or incomplete.

Considering it incomplete is the most charitable interpretation, so we shall proceed under that interpretation. The most likely reason it is incomplete is because the author elided the detail that pending information would also be included in the ledger for simplicity or simply failed to consider this additional detail or simply thought it was implied that referable blocks could not be elided.

Ultimately the number of pending blocks are they can go down over time, so they are a very minor factor regarding ledger pruning.

I think the best thing we can do regarding this information is update the roadmap documentation to be more specific.

Loading

@anarkrypto
Copy link

@anarkrypto anarkrypto commented Sep 12, 2018

If a spammer wants to increase the ledger, to circumvent the pruning he just needs to transact for millions of different accounts, correct?

Loading

@funkspock
Copy link

@funkspock funkspock commented Dec 26, 2018

@kaiquenunes23 account creation and transaction creation is controlled/limited by PoW.

@rkeene Ledger pruning will prune off blocks other than the frontier (head block) +1 or 2 previous for each account. Which to my understanding [Limits history of transactions]

  1. In addition to this, will pruning take place (periodically?) for accounts below a certain threshold (dust accounts)? or like other coins require that "dust accounts" be topped first before they can be used in transactions (for example XRP requires a base reserve of 20). [Limits number of accounts]

  2. Regarding "dust transactions", will these be rate-limited or blocked below a certain threshold (can this threshold be dynamic based on network conditions)? [Limits number of transactions]

Loading

@jmmcgee jmmcgee mentioned this issue Jan 26, 2019
Open
@zhyatt zhyatt removed this from the V19.0 milestone Feb 12, 2019
@zhyatt zhyatt added this to the V20.0 milestone Feb 12, 2019
@zhyatt zhyatt removed this from the V20.0 milestone Mar 15, 2019
@zhyatt zhyatt added this to the Research for Future Release milestone Mar 15, 2019
@lucaslopes
Copy link
Contributor

@lucaslopes lucaslopes commented Oct 19, 2019

  1. Why limit history of transactions is something bad? The relevance of the ledger is its current state, if someone cares about history of one or several wallets, it should be up to them decide to store it, but doesn't having it should not compromise the functionality of the protocol.

  2. Regarding pending blocks: I agree with @Omarino99 in doesn't requiring signatures in receive blocks, this could bring other implications but maybe bring better positive gains to the network. Dividing the account's chain in two (one for sends and other to receives) could solve this problem?

Loading

@arranHarty123
Copy link

@arranHarty123 arranHarty123 commented Dec 1, 2019

Should this be configurable in the node config so instead of just frontier and previous you can choose frontier + x previous blocks (x min 1).

Might give a middle ground between full history and minimal.

Loading

@SergiySW SergiySW mentioned this issue Aug 13, 2020
Merged
@zhyatt zhyatt mentioned this issue Oct 16, 2020
Open
SergiySW added a commit that referenced this issue Nov 6, 2020
@SergiySW @wezrule
Experimental ledger pruning (#2881)
8e5f426 
This is the initial, experimental implementation of pruning, related to #1094. Node ledger pruning is designed to safely discard parts of the ledger. Now ledger size can be reduced from 23.8 GB to 7.5GB after vacuum. And from 18.8 GB to 5.2 GB after database rebuild.

Configuration and considerations
- This version of ledger pruning is for experimental use only:
- Enable optional ledger pruning with node CLI `--debug_prune` for manual pruning & node launch flag `--enable_pruning` for automatic node background actions.
- Pruning rules can be configured in [node.experimental] config with “max_pruning_age” (Time in seconds to limit for blocks age after pruning. Also period for automatic node pruning task, default 1 day, 5 minutes for beta network) & “max_pruning_depth” (Limit for full blocks in chain after pruning, default 0 - unlimited). By default all confirmed blocks older than 1 day can be pruned, except genesis block & last confirmed block in each account chain.
- Last confirmed block in account chain & all unconfirmed blocks cannot be pruned from ledger. Pruning is relying on confirmation height, so confirmation height reset or too low online_weight_minimum config can cause issues.
- Pruned blocks hashes are located in database table “pruned” & can be used by node internally (#2946).

RPC, CLI and other changes
- Several RPC calls are modified to support pruned blocks related to main request block (#2977).
- RPC “history”: if previous block is pruned, “amount” field is not returned for all blocks, “account” field is not returned for state blocks, also state block type/subtype is “unknown” . If source is pruned, then “account” field is not returned for receive/open legacy & for receive/open state blocks (#2977).
- Same as above for QT wallet history (#2977).
- RPC “block_info”/“blocks_info”: if previous block is pruned, “amount” field is not returned. RPC “block_count”: for nodes with enabled pruning returns 2 new fields: “full” blocks & “pruned” blocks. “full” + “pruned” = total “blocks” (#2977).
- Block count for CLI/RPC/telemetry is sum of full blocks & pruned blocks (#2977).
- RPC "pruned_exists" to chekc pruned block existence in local database.

Other notes
- Node is aggressively pruning ledger during initial bootstrap (until hardcoded block count) to limit node disk space usage, pruning config settings are ignored. If you want to avoid this behavior, finish full bootstrap before pruning start.
-  `--enable_pruning` flag & config option “enable_voting” are mutually exclusive to prevent representatives start with pruned ledger. Pruning is not available for any representatives (#2947).
- Pruned node can serve as bootstrap server for remaining blocks in ledger (#2976).
- Multiple internal functions are modified to use pruned blocks in block processor (#2974), internal node wallet actions (#2992), confirmation height processor (#2978) and rep crawler (#2975).
- Development: consider using ledger safe functions for code with possible pruned blocks (#2968).

Additional updates being reviewed
- Include pruned count in telemetry.
- Improve RocksDB pruned block count calculation.

Co-authored-by: Wesley Shillingford <650038+wezrule@users.noreply.github.com>
Co-authored-by: Guilherme Lawless <guilherme@nano.org>
@shamoons
Copy link

@shamoons shamoons commented Dec 3, 2021

Is this still something we're shooting for?

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
Research for Future Release
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
@shamoons @zhyatt @rkeene @lucaslopes @anarkrypto @funkspock @cryptocode @arranHarty123