Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow maintaining the blockfilterindex when using prune #15946

Merged
merged 6 commits into from Feb 18, 2021

Conversation

@jonasschnelli
Copy link
Contributor

@jonasschnelli jonasschnelli commented May 3, 2019

Maintaining the blockfilterindexes in prune mode is possible and may lead to efficient p2p based rescans of wallets (restore backups, import/sweep keys) beyond the prune height (rescans not part of that PR).

This PR allows running the blockfilterindex(es) in conjunction with pruning.

  • Bitcoind/Qt will shutdown during startup when missing block data has been detected ([re]enable -blockfilterindex when we already have pruned)
  • manual block pruning is disabled during blockfilterindex sync
  • auto-pruning is delayed during blockfilterindex sync

ToDos:

  • Functional tests
@DrahtBot
Copy link
Contributor

@DrahtBot DrahtBot commented May 3, 2019

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #20827 (During IBD, prune as much as possible until we get close to where we will eventually keep blocks by luke-jr)
  • #20664 (Add scanblocks RPC call by jonasschnelli)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Loading

src/index/base.cpp Show resolved Hide resolved
Loading
src/index/base.cpp Show resolved Hide resolved
Loading
src/index/base.h Outdated Show resolved Hide resolved
Loading
src/rpc/blockchain.cpp Outdated Show resolved Hide resolved
Loading
src/rpc/blockchain.cpp Outdated Show resolved Hide resolved
Loading
src/validation.cpp Outdated Show resolved Hide resolved
Loading
test/lint/lint-circular-dependencies.sh Show resolved Hide resolved
Loading
@promag
Copy link
Member

@promag promag commented May 6, 2019

Closes #15867?

Loading

@jonasschnelli jonasschnelli force-pushed the 2019/05/prune_blockfilter branch 3 times, most recently from b9537bf to 011904a May 9, 2019
@jonasschnelli
Copy link
Contributor Author

@jonasschnelli jonasschnelli commented May 9, 2019

Rebased and fixed reported points.

Loading

src/index/base.cpp Outdated Show resolved Hide resolved
Loading
src/index/base.cpp Outdated Show resolved Hide resolved
Loading
src/index/base.cpp Outdated Show resolved Hide resolved
Loading
src/index/base.cpp Outdated Show resolved Hide resolved
Loading
src/rpc/blockchain.cpp Outdated Show resolved Hide resolved
Loading
src/validation.cpp Outdated Show resolved Hide resolved
Loading
@laanwj laanwj added the Feature label Sep 30, 2019
@Roasbeef
Copy link

@Roasbeef Roasbeef commented Jan 8, 2020

Any chance this will be revived? This is a blocker for lnd to support a pruned bitcoind node, as we want to be able to fetch filters from bitcoind, then manually fetch blocks ourselves (if bitcoind doesn't have them) to scan them for rescans or just normal wallet sync.

Loading

@prusnak
Copy link
Contributor

@prusnak prusnak commented Jul 8, 2020

This feature increases the value of running a pruned node significantly. +1 for adding this!

Loading

@luke-jr
Copy link
Member

@luke-jr luke-jr commented Jul 8, 2020

I suspect #19463 could be useful for this

Loading

@andronoob andronoob mentioned this pull request Jul 28, 2020
4 tasks
@jonasschnelli jonasschnelli force-pushed the 2019/05/prune_blockfilter branch 3 times, most recently from c49aa4a to 6c7b84c Dec 10, 2020
@jonasschnelli
Copy link
Contributor Author

@jonasschnelli jonasschnelli commented Dec 10, 2020

Rewrote this PR and tried to fix the reported point by @promag, @MarcoFalke and @luke-jr.
Added -fastprune debug parameter to test pruning more effectively (added a functional-test as well).

Loading

@jonasschnelli
Copy link
Contributor Author

@jonasschnelli jonasschnelli commented Feb 11, 2021

Thanks a lot for testing @ryanofsky.
Addressed your points (+rebased)(force pushed).

Loading

@jonasschnelli jonasschnelli force-pushed the 2019/05/prune_blockfilter branch from 0280274 to e2251df Feb 11, 2021
Copy link
Contributor

@ryanofsky ryanofsky left a comment

Code review ACK e2251df. Only change is rebase and implementing some suggestions.

Please consider implementing the suggestion below. It is a small 2 line change which I think would make the non-manual prune code less confusing, more efficient, and more consistent with the manual prune code.

Loading

} else {
LOG_TIME_MILLIS_WITH_CATEGORY("find files to prune", BCLog::BENCH);

m_blockman.FindFilesToPrune(setFilesToPrune, chainparams.PruneAfterHeight(), m_chain.Height(), IsInitialBlockDownload());
m_blockman.FindFilesToPrune(setFilesToPrune, chainparams.PruneAfterHeight(), last_prune, IsInitialBlockDownload());
Copy link
Contributor

@ryanofsky ryanofsky Feb 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "Avoid pruning below the blockfilterindex sync height" (14c9508)

Pretending last_prune as the chain height may be safe, because the only side effect is to not prune blocks that could be pruned, but it is confusing, and also creates a difference in behavior between the manual prune case which uses the real height and this case which pretends the min index sync height is the height.

If this behavior is going to be kept, there should be an explanatory comment because it looks broken, and not obviously broken in a safe way instead of a dangerous way.

But better than adding a comment would be to make the code straightforward: Stop passing last_prune as chain_tip_height. Instead, continue passing m_chain.Height() as chain_tip_height, pass last_prune as prune_height, and assign nLastBlockWeCanPrune = min(prune_height, chain_tip_height - MIN_BLOCKS_TO_KEEP) inside FindFilesToPrune. This would be more consistent with the logic in FindFilesToPruneManual, as well as more efficient, and more obviously correct.

Loading

Copy link
Contributor Author

@jonasschnelli jonasschnelli Feb 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. That makes sense. Implemented now as suggested by @ryanofsky.

Loading

Copy link
Contributor

@fjahr fjahr left a comment

Code review ACK e2251df

I agree with @ryanofsky that his suggestion would be an improvement, at least a comment makes sense and the update to the test description noted below would be good. If you decide to make these changes I will do my best to re-review quickly.

Loading

# Copyright (c) 2020 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
"""Test RPC misc output."""
Copy link
Contributor

@fjahr fjahr Feb 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in b5c243c:

Description needs an update.

Loading

Copy link
Contributor Author

@jonasschnelli jonasschnelli Feb 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for spotting. Fixed.

Loading


def run_test(self):
# test basic pruning compatibility & filter access of pruned blocks
self.log.info("check if we can access a blockfilter when pruning is enabled but no actual pruned blocks")
Copy link
Contributor

@fjahr fjahr Feb 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in b5c243c:

nit: "...but no blocks are actually pruned"

Loading

Copy link
Contributor Author

@jonasschnelli jonasschnelli Feb 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

Loading

Loading
@jonasschnelli jonasschnelli force-pushed the 2019/05/prune_blockfilter branch from e2251df to 84716b1 Feb 16, 2021
@jonasschnelli
Copy link
Contributor Author

@jonasschnelli jonasschnelli commented Feb 16, 2021

Thanks @ryanofsky and @fjahr! Fixed the reported points. Thanks for a quick retest.

Loading

Copy link
Contributor

@ryanofsky ryanofsky left a comment

Code review ACK 84716b1. Only changes since last review were suggested new FindFilesToPrune argument and test.

Change is neat and straightforward and I think close to ready for merge.

This should also get release notes at some point since it adds a new feature and a new error condition that wasn't checked before.

Loading

Copy link
Contributor

@benthecarman benthecarman left a comment

tACK 84716b1

Loading

@fjahr
Copy link
Contributor

@fjahr fjahr commented Feb 17, 2021

Code review ACK 84716b1

Checked that only changes since last review were small improvements on the functional test and the added parameter to FindFilesToPrune().

Loading

@@ -17,6 +17,7 @@
#include <cuckoocache.h>
#include <flatfile.h>
#include <hash.h>
#include <index/blockfilterindex.h>
Copy link
Member

@jnewbery jnewbery Feb 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this making blockfilterindex part of validation code?

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet