multiwallet: issues arising from copying wallet files

[dooglus]

I tried loading up all my wallet files at once to see how well it worked, then ran 'getbalance' on each of them. The balances of some were reported as being zero when they should have been non-zero.

I believe the root cause is that I loaded up some old backups of the funded wallets as well as the up to date versions.

Is that behavior meant to be supported? Or should each wallet have been created separately?

My guess is that the wallet database gets some kind of unique ID assigned to it when it is created, and loading up an old backup alongside the current version means I have two wallets with the same "unique" ID loaded at once, causing problems.

As an example, I have two wallets: wallet.dat.finex and wallet.dat.finex2. One is an old copy of the other. I loaded up a single third wallet, let it sync 3 blocks, shut down, then loaded the two finex wallets together. I had added debug to see why rescanning was being skipped in some cases.

wallet.cpp has this test to decide whether the rescan is necessary:

if (chainActive.Tip() && chainActive.Tip() != pindexRescan)

In the debug output I saw:

2017-09-30 16:32:36 [wallet.dat.finex] rescan? checking tip()
2017-09-30 16:32:36 [wallet.dat.finex] rescan? maybe: chainActive.Tip() is true
2017-09-30 16:32:36 [wallet.dat.finex] rescan? yes: 0x7f60bf3f3db0 (487616) != 0x7f60d3d4aad0 (487613)

2017-09-30 16:32:36 [wallet.dat.finex2] rescan? checking tip()
2017-09-30 16:32:36 [wallet.dat.finex2] rescan? maybe: chainActive.Tip() is true
2017-09-30 16:32:36 [wallet.dat.finex2] rescan? no: 0x7f60bf3f3db0 (487616) == 0x7f60bf3f3db0 (487616)

so the finex wallet gets 3 blocks rescanned, but the finex2 wallet doesn't, even though it needs a rescan. I guess they share a CWalletDB walletdb(*walletInstance->dbw);.

I stopped looking into it here, because maybe the solution is going to be "stop loading copies of a wallet at the same time".

[jonasschnelli]

Hmm... I hope someone can verify that.
But it could indeed be a BDB issue. BerkleyDB operates on a directory & a file. Maybe this is another reason to either have a directory / BDB env per wallet or just finally make some progress for a new database format.

[dooglus]

Here's a simple way to reproduce the problem:

$ ls -l ~/.bitcoin/wallet.dat.x1
ls: cannot access '/home/chris/.bitcoin/wallet.dat.x1': No such file or directory
$ ls -l ~/.bitcoin/wallet.dat.x2
ls: cannot access '/home/chris/.bitcoin/wallet.dat.x2': No such file or directory
$ btc -wallet=wallet.dat.x1 &
$ bc getblockcount
488234
$ bc stop
Bitcoin server stopping
$ cp wallet.dat.x1 wallet.dat.x2
$ btc -wallet=wallet.dat.x3 &
$ sleep 300 [or whatever]
$ bc getblockcount
488235
$ bc stop
Bitcoin server stopping
$ btc -wallet=wallet.dat.x1 -wallet=wallet.dat.x2 -printtoconsole | grep -i rescan
2017-10-04 03:57:31 [wallet.dat.x1] rescan? checking tip()
2017-10-04 03:57:31 [wallet.dat.x1] rescan? maybe: chainActive.Tip() is true
2017-10-04 03:57:31 [wallet.dat.x1] rescan? yes: 0x7f0cb341d450 (488235) != 0x7f0cca494000 (488234)
2017-10-04 03:57:31 init message: Rescanning...
2017-10-04 03:57:31 [wallet.dat.x1] Rescanning last 1 blocks (from block 488234)...
2017-10-04 03:57:31 [wallet.dat.x1]  rescan                   25ms
2017-10-04 03:57:31 [wallet.dat.x2] rescan? checking tip()
2017-10-04 03:57:31 [wallet.dat.x2] rescan? maybe: chainActive.Tip() is true
2017-10-04 03:57:31 [wallet.dat.x2] rescan? no: 0x7f0cb341d450 (488235) == 0x7f0cb341d450 (488235)
$ 

• show that x1 and x2 don't exist
• create x1 and run getblockcount
• copy x1 to x2
• run new wallet x3 until a new block is found
• run wallets x1 and x2, and note that only x1 is rescanned while both x1 and x2 require a 1 block rescan

[ryanofsky]

I was able to reproduce this, and I think I can add better error handling to detect it, but it doesn't appear to be fixable because Berkeley db can't distinguish between the two copies of the wallet file. Basically your idea about wallet databases having "some kind of unique ID" is right. From https://docs.oracle.com/cd/E17275_01/html/programmer_reference/program_copy.html:

Because system file identification information (for example, filenames, device and inode numbers, volume and file IDs, and so on) are not necessarily unique or maintained across system reboots, each Berkeley DB database file contains a unique 20-byte file identification bytestring. When multiple processes or threads open the same database file in Berkeley DB, it is this bytestring that is used to ensure the same underlying pages are updated in the database environment cache, no matter which Berkeley DB handle is used for the operation.

What happens as a result is that after WriteBestBlock is called on the first wallet file following the rescan, the ReadBestBlock call on the second wallet picks up the not fully flushed value from the first wallet, instead of the right value inside its own wallet file.

We can retrieve the file id's with pdb->get_mpf()->get_fileid(), so it should be possible to add a check that will refuse to load more than one wallet with the same id. I'll try to open a PR that does this today.

[dooglus]

Thanks @ryanofsky. I'm glad you took the time to look into this. I'm guessing the bug could result in wallet file corruption if two wallet files are sharing a single database cache, so it's good to fix and backport.