"last wallet synchronisation goes beyond pruned data" on totally new pruned node

[gmaxwell]

Coping from: https://bitcointalk.org/index.php?topic=1347286.msg13832507#msg13832507

"I've been testing out 0.12.0rc3 in pruned mode, and have run into the following error message when restarting Bitcoin Core: "last wallet synchronisation goes beyond pruned data. You need to -reindex (download the whole blockchain again in case of pruned node)"

To reproduce:
0.) Start from scratch, deleting the entire Bitcoin data directory if it exists (leaving no wallet file or blockchain data).
1.) Install Bitcoin Core
2.) Create Bitcoin data directory if it doesn't exist, create bitcoin.conf, set "prune=550" without quotes
3.) Run Bitcoin Core
4.) After you've verified, say, 100,000 blocks, close Bitcoin Core. Wait until the warning window to not shutdown/restart computer completely closes.
5.) Run Bitcoin Core again, let it verify some more, close it again.
6.) Rinse and repeat step 5...

At some point, I get the above error message on starting Bitcoin Core.

I don't know what exactly is causing this, but if you're like me and you've verified a large amount of blocks and then you're told that you have to start completely over, you'll probably be pissed.

This error message is not new, as a related fix for this kind of error, when converting non-pruned nodes to pruned nodes was made here: #6345

While it's obviously easy to reproduce this error if you try plopping an already existing wallet.dat file into a pruned node data directory and then running Core -- that's not what I've been experiencing. I've been seeing this error with a brand new wallet.dat that was created from scratch alongside a fresh, pruned node."

[gmaxwell]

please tag wallet.

[mmortal03]

I'm working to attach debug.log and wallet.dat. If you need further verbosity or any other files, let me know, and I'll try to reproduce the error again to produce these for you. It's pretty straightforward to reproduce the issue, but time consuming and doesn't happen every time. It's happened to me enough times in the last few days, though, that I think I'll be able to.

Machine specs are: AMD E-350 Processor, 6 GB RAM, Windows 10 Pro, 64-bit.
Bitcoin Core 0.12.0rc3 64-bit.

bitcoin.conf:
daemon=1
server=1
maxuploadtarget=144
prune=550

[mmortal03]

Here is the debug.log and wallet.dat:
https://dde19c9eff451e7e0b7d819897f53dd506692a8f.googledrive.com/host/0B2Wm1faUufb7N3J5QkRFeHpRUkk/Bitcoin_prune_debug.zip

[laanwj]

Haven't looked at the wallet and debug.log yet, but at first glance it appears that when it is blasting though large number of blocks (e.g. during initial sync, or when catching up) it is possible for the wallet (at least the state synced to disk, even on a clean shutdown) to lag behind more than the amount of blocks that is retained.

[laanwj]

From that wallet.dat:

bestblock 0000000000000000006b500f79750a2c5a48da5ad2a13fed6cdaffade6cada25 ...

So block 373499. According to the debug.log the chain was way ahead of that before shutting down for the last time:

2016-02-10 00:11:27 UpdateTip: new best=000000000000000004fd3098365fb488f31f013ee41d0de63bdb7b4b0aa1dcb2  height=373816  log2_work=83.32589  tx=83020932  date=2015-09-10 04:35:03 progress=0.854354  cache=20.6MiB(12251tx)

(373816 -373499)=317 blocks to be exact.

[sipa]

It seems that the bestblock descriptor in the wallet is not updated when flushing occurs for pruning, which is a bug. It could explain the issue here, but only in the case of an unclean shutdown (as the bestblock descriptor is always updated on clean shutdown).

[laanwj]

As the debug.log does show a clean shutdown, I hope this doesn't mean that there are problems with clean shutdown not being as clean as we think. But updating the bestblock descriptor when flushing for pruning would probably solve the immediate issue.

[morcos]

@laanwj It looks to me like a clean shutdown still doesn't update the bestblock descriptor if it has been less than an hour. So I think this problem could happen with clean shutdowns but will be fixed properly by #7502.

[sipa]

@morcos Nice catch. I think that explains it.

[laanwj]

@laanwj It looks to me like a clean shutdown still doesn't update the bestblock descriptor if it has been less than an hour

Yes that must be the problem then :(

Solved by #7502 (which is in rc5)