Prune locks

[luke-jr]

This adds the ability to have any number of internal or external locks to prevent pruning specific block ranges.

Some examples where this would be useful:

• Ensuring any known Core wallet won't become unable to sync due to pruning when it is unloaded. (Included in this PR)
• Ensuring pruning doesn't go too far for a desired-functional wallet backup. (Questionable, since you probably lose the node if you need the backup anyway?)
• Allowing users to temporarily disable block indexes and filters, without pruning cutting them off from catching up later.
• Avoiding pruning out from under external wallets (Armory, EPS, etc) that might need block data.
• External block indexes/filters, etc.

By design, users retain complete control over prune locks, and can delete them out from under applications using them. This is to avoid circumstances where a prune lock has been set, by an application that may no longer be used.

[DrahtBot]

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept ACK	fjahr

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #30342 (kernel, logging: Pass Logger instances to kernel objects by ryanofsky)
• #29668 (prune, rpc: Check undo data when finding pruneheight by fjahr)
• #29652 (wallet: Avoid potentially writing incorrect best block locator by ryanofsky)
• #29418 (rpc: provide per message stats for global traffic via new RPC 'getnetmsgstats' by vasild)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

[andronoob]

I think redownloading specified block could be much more useful, just like the case that scantxoutset gives the corresponding block height, but the actual block had already been pruned - the ability to redownload (and then rescan) specified block fix this problem.

[andronoob]

Recently I heard (could be inaccurate/misunderstanding of myself) that some projects like BTCPay didn't keep full transaction data of received funds. Only the UTXO entry (txid, vout, amount and scriptPubkey) was kept. Therefore they are facing problems to fix the BIP143 vulnerability, because the full data of the previous transaction had been already discarded. I think the ability to redownload & rescan specified block(s) can relieve this problem.

[luke-jr]

I think redownloading specified block could be much more useful, just like the case that scantxoutset gives the corresponding block height, but the actual block had already been pruned - the ability to redownload (and then rescan) specified block fix this problem.

That's useful too, but for different use cases. You wouldn't want to be constantly redownloading the same block over and over. (Prune locks might be useful for it, though, to prevent the redownloaded block from being immediately repruned before use.)

Recently I heard (could be inaccurate/misunderstanding of myself) that some projects like BTCPay didn't keep full transaction data of received funds. Only the UTXO entry (txid, vout, amount and scriptPubkey) was kept.

Aside, that's a pretty serious bug. It's the receiver's responsibility to rebroadcast the transaction until it confirms...

[andronoob]

Allowing users to temporarily disable block indexes and filters, without pruning cutting them off from catching up later.

Honestly I don't get what's the point of this - to save a little more disk space under extreme conditions, like almostly exhausted disk space, so that the node could then keep running for a littel more time?

External block indexes/filters, etc.

This seems to justify the above point, however I think one of the most-wanted features is "external (and pluggable) block storage", rather than "external indices/filters etc", although the later also seems to be nice (despite both would complicate the things further).

Avoiding pruning out from under external wallets (Armory, EPS, etc) that might need block data.

Ensuring any known Core wallet won't become unable to sync due to pruning when it is unloaded. (Included in this PR)

Then, (without some filter/matching mechanism) it will in fact disable (pause) pruning until the wallet reloads, won't it? Because the node won't know whether a newly received block is unrelated (prunable), therefore the only choice is to stop pruning.

It has been a significant usability problem since quite a long time ago, that the user cannot load/import wallets/keys/addrs freely - nobody likes triggering the troubles of blockchain rescanning.

I think #15946 (together with something like #15845) should be able to fix such usability problem in a simpler,better way, that:

With local blockfilterindex, the blocks could be simply discarded (pruned). Then, the once pruned blocks could be redownloaded according to local blockfilterindex while reloading wallet/importing keys/addrs - no need to worry about whether a block could be needed in the future anymore.

However I still agree that a button allowing the user to simply "pause/resume" pruning would be nice, too - I hope that this could be displayed on some dashboard.

[andronoob]

That's useful too, but for different use cases. You wouldn't want to be constantly redownloading the same block over and over. (Prune locks might be useful for it, though, to prevent the redownloaded block from being immediately repruned before use.)

Yes, prune locks indeedly have its utility, especially to lock needed/related historical blocks. I just think that I see greater problems.

[luke-jr]

Rebased

[DrahtBot]

If still relevant, this needs rebase due to silent merge conflict.

[DrahtBot]

Converted to draft due to failing CI and inactivity

[DrahtBot]

🐙 This pull request conflicts with the target branch and needs rebase.

[DrahtBot]

⌛ There hasn't been much activity lately and the patch still needs rebase. What is the status here?

• Is it still relevant? ➡️ Please solve the conflicts to make it ready for review and to ensure the CI passes.
• Is it no longer relevant? ➡️ Please close.
• Did the author lose interest or time to work on this? ➡️ Please close it and mark it 'Up for grabs' with the label, so that it can be picked up in the future.