[Wallet] Replace -rescan with a new RPC call "rescanblockchain"

[jonasschnelli]

A RPC rescan command is much more flexible for the following reasons:

• You can define the start and end-height
• It can be called during runtime
• It can work in multiwallet environment

[laanwj]

I'd rather have it that the API is such that an explicit rescan is never needed. Wasn't there some work on a multi-import w/ timestamps?

[jonasschnelli]

Yes. There is a PR (see PR description). I agree that it would be better to avoid rescans at all, although it might be complicated to catch all edge-cases and a manual trigger can help in situations where someone needs to deal with multiple/complex imports (you only want to do one rescan).

And I think some people will cancel a rescan because they want to do other stuff and/or had not considered that a rescan can take a long time. Afterward calling -rescan (from genesis) is a time consuming option.

[laanwj]

But manually specifying a block # to rescan from is extremely fragile... it's very easy to get this wrong.

Also, rescanning doesn't interact with pruning which will be more and more common in the future.

[gmaxwell]

@lannwj I thought thats part of what the height parameter here was for-- addressing pruning comparability?

[petertodd]

How about we call this rescanfromheight instead, to make it possible to add a rescanfromtime later if users demand? Equally, some kind of RPC call that finds the first block with a nTime after a specific time might be useful here.

Do we have a way of querying what block # is the oldest non-pruned block?

[petertodd]

It also occurs to me that for this usecase we might instead want to have pruning not happen automatically, but rather be an on-demand thing where the user specifies the oldest time they're interested in.

[gmaxwell]

So the biggest negative I personally see here is that it furthers this misunderstanding that rescan is some thing users generally need to be doing. Until we added these non-rescan imports a user initiated rescan is something that never should have been needed (and indicated a serious bug we'd like to know about if it was). As a result of the -rescan argument there is now this whole cargo cult of people that rescan every time they're scarred by a shadow. I hate to further that.

But I can't deny how really useful this will be.

[petertodd]

@gmaxwell A possible way around that would be to make the rescan check if you have any addresses that haven't yet been scanned in that range and error out if not. (basically make it say "no rescan needed")

[gmaxwell]

Hm. this could also take a stop argument, allowing you to scan single blocks or avoid rescan overlap. Also, I think all the wallet re-scanning should traverse its interval backwards-- for more instant gratification; though this would dork with the wallet transaction ordering... actually import at all breaks that, I should go talk to luke-jr about that.

[promag]

Remove?

[jonasschnelli]

Oops. A rebase issue. Thanks for point out. Fixed.

[pstratem]

agree with gmaxwell that this should scan a range of blocks

[jonasschnelli]

Agree with the stop parameter. Working on a implementation....

[sipa]

I would still prefer an approach that imports with birthdate instead of explicit rescanning.

[jonasschnelli]

Added a commit that allows providing a optional parameter with a height where the rescan should stop.

@sipa: I agree that rescan height over a key/address birthday would be nice to have (see #6570). But a explicit rescan RPC call can be useful IMO. It's trivial to maintain and it can save lots of rescan-time on the user side. But agree, it has to be considered as "experts" feature.

What about implementing a threshold for autodetecting wether the parameter is a blockheight or timestamp (similar to LOCKTIME_THRESHOLD)?

[promag]

@jonasschnelli see #6570 (comment) regarding your last comment.

[GIJensen]

ACK

[mrbandrews]

If this is still moving forward - I tested it a bit (including pruned mode) and it looks fine to me. One suggestion is to make the start-height a required parameter so that the user specifies "rescanblockchain 1" to scan from genesis. If the start-height is < 1 or higher than current height, throw an error.
Otherwise, ACK.

[promag]

Move to while condition?

[promag]

Because it's inclusive?

[jonasschnelli]

Rebased.
I think there are still reasons to consider that PR. At the moment, it would really be useful. Even once we have #7551 (importmulti) it could see use cases for rescanblockchain.

[sipa]

This seems better #7984, but I still prefer not furthering the usage of various rescans. We need APIs that don't require users to keep track of the concept of rescanning IMHO.

[jonasschnelli]

I agree. Ideally, there will be no need to rescan. But in practice, rescans are sometimes required (I guess everyone who gave some users support has encountered that). IMO a rpc rescan commend with an optional hight is much more flexible then -rescan as a startup argument.

[sipa]

I think we should work on importmulti instead (or at least an importprivkey that takes a key birthdate as parameter), not on more ways to rescan.

[jonasschnelli]

I think we should work on importmulti instead (or at least an importprivkey that takes a key birthdate as parameter), not on more ways to rescan.

Yes. I can agree with that.

[jonasschnelli]

Another thing where this could be useful is restoring hd wallets

[MarcoFalke]

Since importmulti #7551 is merged, some use cases are covered by that.

[laanwj]

I think we should work on importmulti instead (or at least an importprivkey that takes a key birthdate as parameter), not on more ways to rescan.

Tend to agree. Closing this for now.

[jonasschnelli]

IMO something like that would be very handy if you rescan a HD wallet (old backup).
-rescan does not allow direct user feedback
IMO rescanning an old HD backup should by default start at the height where we introduces HD (optional down to genesis).

[jonasschnelli]

Reopened and overhauled.
This now does replace the -rescan startup argument with a new RPC call rescanblockchain. The reasons for that are:

• You can define the start and end-height
• It can be called during runtime
• It can work in multiwallet environment

Using -rescan will cancel the startup with an error referring to the new RPC call.

[ryanofsky]

Should add some test coverage for this. One easy way might be to add a new Rescan.manual enum value here: https://github.com/bitcoin/bitcoin/blob/46771514fa86b9a5a0e0af34c1abfa1da22212f7/test/functional/import-rescan.py#L32, then put

if self.rescan == Rescan.manual:
    self.node.rescanblockchain()

at the end of the do_import method.

[ryanofsky]

Maybe still worth mentioning that pruning can get in the way of successfully importing wallet keys & rescanning.

[ryanofsky]

Maybe this should take either times or heights like the pruneblockchain RPC: https://github.com/bitcoin/bitcoin/blob/46771514fa86b9a5a0e0af34c1abfa1da22212f7/src/rpc/blockchain.cpp#L838

[jnewbery]

suggest you change the argument names to startheight and stopheight. No other rpcs use - as word delimiter.

[jonasschnelli]

Indeed. Fixed.

[ryanofsky]

Style might need to be updated for this code (moving opening brace to same line here, adding missing braces other lines)

[jonasschnelli]

Fixed.

[ryanofsky]

Might be helpful if error message could mention getblockchaininfo RPC for getting pruned height.

[jonasschnelli]

Fixed.

[ryanofsky]

I think these files are automatically generated from -help output and could be reverted in the PR.

[jonasschnelli]

Fixed.

[ryanofsky]

I don't understand what the use-case is for the stop argument. Can you describe a scenario where you would want to provide it? I can see how it might have been desirable before there was an abortrescan RPC to break a big scan up into little scans, but I don't see why you'd want it now.

[jonasschnelli]

See the discussion about the stop argument: #7061 (comment)

[ryanofsky]

Is there any advantage to getting rid of the -rescan option if we still have keep all the rescan logic below? Is there a future cleanup or new feature that will be easier to implement with the option gone?

[jonasschnelli]

The -rescan option is global while this PR changes it (partially) to per-wallet. Still, -zapwallettx, etc. need to also be moved to per-wallet basis (I don't know how right now).

But removing the global -rescan option has to be done sooner or later if we want proper multiwallet.

[ryanofsky]

But removing the global -rescan option has to be done sooner or later if we want proper multiwallet.

Do -zapwallettx, etc also need to be removed to support proper multiwallet?

[jonasschnelli]

IMO Yes. It's pure db utility. Either we move it to RPC or to a new wallet/db tool.

[jnewbery]

This is confusing: the if statement requires !needsRescan. Shouldn't that be the other way around? If needsRescan is true, we should rescan. I think the if statement should be:

if (needsRescan || GetBoolArg("-salvagewallet", true) || GetBoolArg("-zapwallettxes", true))

re: zapwallet and salvagewallet. I agree that they should be changed to RPCs for multiwallet.

[jonasschnelli]

No. I don't think so.
If that if statement is true, we load the wallet best block which prevents a complete rescan (so it's the opposite).
So we only load the wallet best block (== no rescan) if needsRescan is false and -salvagewallet and --zapwallettxes` have not been set.

[jnewbery]

Yes, you're right.

[jnewbery]

nit: update function comment to say that the rescan is up to pindexStop if it is non-null

[jonasschnelli]

Fixed.

[jnewbery]

Need to remove more of this comment, so that it just reads:

// show rescan progress in GUI as dialog

[jonasschnelli]

But can't it also happens during startup when the GUI will show the progress only in the splash-screen and not in a dialog?

[jnewbery]

This is confusing: the if statement requires !needsRescan. Shouldn't that be the other way around? If needsRescan is true, we should rescan. I think the if statement should be:

if (needsRescan || GetBoolArg("-salvagewallet", true) || GetBoolArg("-zapwallettxes", true))

re: zapwallet and salvagewallet. I agree that they should be changed to RPCs for multiwallet.

[jnewbery]

suggest you change the argument names to startheight and stopheight. No other rpcs use - as word delimiter.

[jnewbery]

This should throw an error if the argument isn't a number, rather than continue. It should allow the argument to be Null. I think the if test you need is:

if (request.params.size() > 0 && !request.params[0].isNull()) {

[jnewbery]

should this return an error if heightstop is lower than heightstart? If heightstop is higher than the tip height?

[jnewbery]

This RPC should return a message to the user if it is successful, eg "Blockchain rescanned from block <hash> height <height> to block <hash> height <height>. <numtx> transactions added to wallet". Otherwise it's impossible for the user to know whether the call was successful or not.

[ryanofsky]

Should at least throw an error this isn't successful (the ScanForWalletTransactions call will return null on success, otherwise a pointer to the last failing block, so this should be pretty easy).

[jnewbery]

I can't see where rescanblockchain() is being called. Am I missing something?

[jonasschnelli]

It happens outside of this call (somewhere in wallet.cpp). But the comment doesn't say "It" calls rescanblockchain, it either says "you" can call rescanblockchain().

[jnewbery]

I don't understand. The comment is:

    // Recovery procedure:
    // move wallet file to wallet.timestamp.bak
    // Call Salvage with fAggressive=true to
    // get as much data as possible.
    // Rewrite salvaged data to fresh wallet file
    // call rescanblockchain (RPC) so any missing
    // transactions will be found.

This function does:

• move wallet file to wallet.timestamp.bak
• Call Salvage with fAggressive=true to get as much data as possible.
• Rewrite salvaged data to fresh wallet file

but it doesn't call rescanblockchain, and I can't see rescanblockchain being called by the callers.

[jonasschnelli]

Yes. You right. The rescan is not part of the call. But the part that calls CDB::Recover does always call a rescan. But now I got your point. We should refer to ScanForWalletTransactions() instead to rescanblockchain (RPC). Right?

[jnewbery]

sorry, I'm still not seeing it. You say "the part that calls CDB::Recover does always call a rescan". Where?

[jonasschnelli]

It's confusing. But CDB::Recover only gets calls by setting -salvagewallet which does set -rescan. Maybe I should remove that comment part... I don't know what's best because -rescan is currently mentioned there.

[jnewbery]

ah, makes sense now. Thank you. I wasn't looking in init.cpp, but now I think I see how this fits together. CWallet::Verify() is called first, which calls CWalletDB::Recover() if -salvagewallet is set. Later in AppInitMain(), we call CWallet::AppInitMain(), which is where the rescan happens if -salvagewallet is set.

Perhaps change the comment to something like:

    // Try to recover the wallet:
    // - move wallet file to wallet.timestamp.bak
    // - call Salvage with fAggressive=true to get as much data as possible.
    // - rewrite salvaged data to fresh wallet file
    //
    // CWallet::AppInitMain() will later run a full blockchain rescan to find
    // any missing transactions.

[TheBlueMatt]

This needs rebase (probably just wait till after 15 at this point). I think outstanding objections to the idea have all largely been removed, Concept ACK from me, at least.

[jonasschnelli]

Rebased.

[ryanofsky]

utACK 7858b9f, but I would prefer if this just deprecated the -rescan option instead of removing it. Removing it with no warning seems needless and kind of jerky, since it barely saves any code.

[ryanofsky]

Capitalization of height doesn't match actual param name.

[ryanofsky]

Could do what pruneblockchain does and allow height to be specified either as a physical height or a time: https://github.com/bitcoin/bitcoin/blob/1caafa6cde3b88d926611771f9b4c06fcc6e0007/src/rpc/blockchain.cpp#L855

[ryanofsky]

Size checks can be dropped here and below. params[0] will return a null value if the param is missing.

[ryanofsky]

This should check against pIndexStop too, so error won't trigger unnecessarily in cases where missing blocks are noncontinuous (which can happen with manual pruning).

[ryanofsky]

Should at least throw an error this isn't successful (the ScanForWalletTransactions call will return null on success, otherwise a pointer to the last failing block, so this should be pretty easy).

[promag]

If #10941 goes first then the test can be extended to do a rescanblockchain and assert the notified blocks.