New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamically Loadable Multiple Wallet Support Complete!!! #2124

Closed
wants to merge 38 commits into
base: master
from

Conversation

Projects
None yet
@CodeShark
Contributor

CodeShark commented Dec 23, 2012

bitcoind now supports loading more than one wallet at once.

A singleton object of type CWalletManager now exists. It handles dynamic loading/unloading and synchronization of wallets and allows different parts of the application to access wallets by name.

A new CWallet* parameter has been added to the RPC functions. Functions which do not use a wallet simply ignore it. In addition, a new field has been added to CRPCCommand that tells us whether or not the function can be called on a wallet.

Wallet-specific information has been removed from RPC method getinfo. Instead, getinfo just reports how many
wallets are currently loaded. Detailed wallet info is now available via the listwallets method.

Four new RPC methods have been added:

  • listwallets
    Returns an array containing wallet information.
  • usewallet [params]
    Allows you to apply existing RPC commands to different wallets.
    A default wallet named the empty string is always loaded and is used if calls are made without usewallet.
    Example: bitcoind usewallet foo listreceivedbyaddress 0 true
    (thanks, gmaxwell, for the idea)
  • loadwallet [rescan=false] [upgradewallet=false] [maxversion=(latest)]
    Dynamically loads an existing wallet file wallet-.dat.
    If no wallet file exists a new wallet is created. The default wallet file is always called wallet.dat.
  • unloadwallet

To specify additional wallets at startup, use option -usewallet=foo -usewallet=bar etc...as detailed here: #2124 (comment) and CodeShark@9d201cf

TODO:

  • Clean up I/O in CWalletManager::LoadWallet - debug, error, and UI output functions.
  • Check synchronization code.
  • Test mining functionality. Allow RPC mining on arbitrary wallets.
  • Integrate with Qt.
@CodeShark

This comment has been minimized.

Contributor

CodeShark commented Dec 23, 2012

It's a work in progress and can certainly be improved. I welcome all comments, suggestions, criticisms, and witty insults.

@Diapolo

This comment has been minimized.

Diapolo commented Dec 23, 2012

Would it be better or possible to use references instead of pointers in your code?

@CodeShark

This comment has been minimized.

Contributor

CodeShark commented Dec 23, 2012

Possible, sure. Better, why?

@CodeShark

This comment has been minimized.

Contributor

CodeShark commented Dec 23, 2012

The sanity test failed because the test suite uses the old RPC function prototype. I was still able to build and run using

  • make -f makefile.unix bitcoind
@rebroad

This comment has been minimized.

Contributor

rebroad commented Dec 23, 2012

This may be an opportunity to move away from the wallets being called "wallet.dat", which makes it all the more easier for malware to steal the contents.

@CodeShark

This comment has been minimized.

Contributor

CodeShark commented Dec 24, 2012

You can now specify additional wallets in the config file or via command line arguments:

usewallet=foo

or

bitcoind -usewallet=foo

A default wallet called "" in the RPC and using file "wallet.dat" is always loaded, as to not break compatibility with the master branch. Therefore, additional wallets should neither be called "default" nor "wallet".

The wallet will be stored in a file called wallet-foo.dat. If the wallet doesn't yet exist, it will be created the first time the wallet is loaded.

CodeShark added some commits Dec 21, 2012

Added multiple wallet support
=============================

A new global structure called pWalletMap has been added. Upon initialization, multiple wallets can be loaded.

A new CWallet* parameter has been added to the RPC functions. Functions which do not use a wallet simply ignore it. In addition, a new field has been added to CRPCCommand that tells us whether or not the function uses a wallet.

Two new RPC methods have been added:
  1) listwallets - Returns an array containing wallet names.
  2) usewallet - Prefix existing wallet RPC calls with usewallet <walletname> to use a particular wallet. If no wallet is specified, the default wallet is used.
    example: bitcoind usewallet foo listreceivedbyaddress 0 true

TODO:
  1) Get wallet names from bitcoin.conf in init.cpp.
  2) Change help RPC so that the message does not depend on wallet state (i.e. walletpassphrase RPC call)
Added loading of wallet names and files from -usewallet arguments
=================================================================

You can now specify additional wallets in the config file or via command line arguments:

	usewallet=foo

or

	bitcoind -usewallet=foo

A default wallet called "default" in the RPC and using file "wallet.dat" is always loaded, as to not break compatibility with the master branch. Therefore, additional wallets should neither be called "default" nor "wallet" nor anything else that might conflict with other filenames in the .bitcoin directory.

The wallet will be stored in a file called foo.dat. If the wallet doesn't yet exist, it will be created the first time bitcoind is run.

TODO: Ensure filename collisions cannot occur. Perhaps allow the wallet files to be arbitrarily named rather than tied to their identifier.
TxMemPool::accept() calls SyncWithWallets(), so all mempool transacti…
…ons are synched automatically without having to call SyncWithWallets() explicitly. This is important when sending between two wallets in a single bitcoind instance so that the receiving wallet is alerted of the transaction immediately.

Removed calls to SyncWithWallets() in ProcessMessage() since SyncWithWallets() is already called by tx.AcceptToMemoryPool().

SyncWithWallets() is still called explicitly from CBlock::ConnectBlock() since the transaction never goes through the mempool in this case.
Removed dependency on wallet encryption state for getting help on RPC…
… methods. Help always returns usage info now.
@CodeShark

This comment has been minimized.

Contributor

CodeShark commented Dec 24, 2012

The getinfo RPC method now returns an array of wallets each with wallet-specific information. This change is, unfortunately, not backwards compatible. However, it doesn't really seem to make sense to make this call wallet-specific. And returning an array of wallets seems to be a very useful feature.

@gmaxwell

This comment has been minimized.

Member

gmaxwell commented Dec 24, 2012

An alternative would be to move that data out of getinfo and into a getwallet info. Part of the reason to do that is that the wallet outputs in getinfo can be rather slow already, and doing them for N wallets won't help matters. (Though lets see if anyone else has an opinion, I could be on drugs here)

@CodeShark

This comment has been minimized.

Contributor

CodeShark commented Dec 24, 2012

I was thinking listwallets should maybe show all this information with a verbose option. Without the verbose option, it would just give a list of the wallet names.

@gmaxwell

This comment has been minimized.

Member

gmaxwell commented Dec 24, 2012

Hm. From the perspective of the RPC as a CLI the use of verbose flags isn't especially discoverable— for something basic like 'get your balances' that is probably worth exposing at the top level. ::shrugs::

@CodeShark

This comment has been minimized.

Contributor

CodeShark commented Dec 24, 2012

What about making the full info the default and having an option to shorten it? That way interactive users get relevant information while application developers seeking to improve performance have an option to do so if they don't need all that information. Of course we can just have two distinct RPC methods...but I'd rather avoid having too many RPC calls when the semantics are similar.

@CodeShark

This comment has been minimized.

Contributor

CodeShark commented Dec 24, 2012

I do think that the wallet information should be completely removed from getinfo, though.

Anyhow, this is the kind of stuff that's really easy to implement and change up front but becomes a serious problem to change once people start using it to build applications. I'd also like to hear several more opinions on this before committing to anything specific.

@CodeShark

This comment has been minimized.

Contributor

CodeShark commented Dec 24, 2012

Another issue I'm wondering about is programmatic access to the config file. We could just append the new wallets to the end of the file whenever they are created - but it would be nice to be able to have tags for sections. Presumably whenever a user creates a new wallet they want it to be loaded at next startup unless they explicitly remove it from the config file.

@mikegogulski

This comment has been minimized.

mikegogulski commented Dec 25, 2012

Multiple wallet support came into my head as I was doing #2075 but I left it aside in favor of trying to move the code in the direction of further modularization improvements between the RPC interface and the wallet and the wallet database as a prerequisite. I put off coding there since I expected something like this to come along and because commentary on the pull request died away.

I can't review the code in detail now but this capability needs to happen. Passing CWallet * around is an improvement for sure, definitely better than another global. I'm finding it kinda cringeworth-brittle from an OO perspective, though. It would be lovely to see new RPC functions implemented in ways that drag suitable amounts of code that depends strongly on the wallet's internal representation out along with them the appropriate changes to the wallet object.

Have you tried merging my code? I'll have to look at that later. Multiple wallets are useful for any number of reasons. Certainly the Bitcoin-only accountants and would-be tax collectors of the future aren't going to be running n instances of bitcoind on their desktops.

Ultimate I think the wallet should be decoupled from the rest of the satoshi client. Pushing standarized interfaces down to wallet-object level is somewhere on the road to getting there. You seem to be addressing some modularization issues in other areas though, too.

@CodeShark

This comment has been minimized.

Contributor

CodeShark commented Dec 25, 2012

As for the OO stuff, I would like to abstract things further and define a listener interface, a transaction store interface, and a key store interface. Technically speaking, a signing agent doesn't even need to store complete transactions. Private keys and unspent outpoints are all it needs. Furthermore, a listening agent could send alerts to other components without the need to store any transactions. For instance, it could just provide filtering.

Having said that, in the interest of seeing these pull requests merged with bitcoin/bitcoin master ASAP, I'm willing to do things incrementally and use whatever structures already exist for now. However, I'd be very interested in giving all this some deeper thought and coming up with a serious reorganization for some later version.

As for now, at the very least it is possible to encapsulate the existing CWallet class much better - and I applaud your efforts.

result = pcmd->actor(params, false);
if (pcmd->unlocked) {
result = pcmd->actor(pWallet, params, false);
}
else {
LOCK2(cs_main, pwalletMain->cs_wallet);

This comment has been minimized.

@sipa

sipa Dec 25, 2012

Member

This will need to lock the appropriate wallet (if any), not pwalletMain.

@@ -25,6 +25,7 @@
using namespace std;
using namespace boost;
CWalletMap* pWalletMap;

This comment has been minimized.

@sipa

sipa Dec 25, 2012

Member

pWalletMap can be allocated statically, I believe.

This comment has been minimized.

@CodeShark

CodeShark Dec 25, 2012

Contributor

A problem is that it is declared in both init.cpp and test_bitcoin.cpp - and these globals must be accessible from rpcwallet.cpp as well, which includes init.h for the externs. I would prefer the wallet RPC calls to be placed in an RPC class which takes a wallet map in its constructor - and then we could remove any dependencies on these externs entirely and restrict the variable scope to init.cpp or the unit test main. But yeah, it could be allocated statically.

public:
wallet_map wallets;
~CWalletMap()

This comment has been minimized.

@sipa

sipa Dec 25, 2012

Member

You may want to use an auto_ptr to make this cleanup happen automatically.

This comment has been minimized.

@CodeShark

CodeShark Dec 25, 2012

Contributor

I was thinking to expand this class to also handle loading of wallets from config parameters.

Perhaps init.cpp could pass io context stuff to the loader method so that the loader method needn't call printf directly.

@sipa

This comment has been minimized.

Member

sipa commented Dec 25, 2012

In general: I'm very much in favor of having native support for multiple wallets. Wallet stuff hasn't been a priority for some time, but if necessary I'd like to help to get this working.

Regarding the idea of separating wallets for block chain processing: sure, that's definitely where we need to go in the future. Ideally, I think the code evolves to separate wallet and the rest along a clean interface in a first step indeed. In a second step, we can maybe make them separate processes started from the same binary or even just separate binaries with shared codebase. The final aim should be entirely separate things, either communicating via some private "wallet interconnect protocol" (where several wallets on a trusted network connect to a single validation service), or even turning the wallet processes into standalone SPV clients (with their own blockheader-chain), that can either connect directly to the network, or can connect to a server provide block/tx validation service, simply via the P2P protocol.

@CodeShark

This comment has been minimized.

Contributor

CodeShark commented Jan 14, 2013

Hmm, still having some issues with the default wallet not unloading at shutdown. Will need to go over the shutdown sequence to make sure all threads relinquish their pointers to it.

@gavinandresen

This comment has been minimized.

Contributor

gavinandresen commented Jan 14, 2013

Just to set expectations:

This big a change, in this critical a part of the codebase, will need:

  • A comprehensive test plan, that exercises all the things that have changed.
  • Testing by at least two different people who aren't developers, on Windows and either Linux or Mac

See https://github.com/bitcoin/QA for the lastest QA process.

@BitcoinPullTester

This comment has been minimized.

BitcoinPullTester commented Jan 24, 2013

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/1c45edce2fbe76f7e146ee12c1df80c66c5fa8f9 for binaries and test log.

@BitcoinPullTester

This comment has been minimized.

BitcoinPullTester commented Sep 16, 2013

Automatic sanity-testing: FAILED MERGE, see http://jenkins.bluematt.me/pull-tester/6c7f86ae371f955371fba05c7f747e4659cd39c8 for test log.

This pull does not merge cleanly onto current master
This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/
Contact BlueMatt on freenode if something looks broken.

@MrLei

This comment has been minimized.

MrLei commented Oct 4, 2013

How i can install bitcoind whit Dynamically Loadable Multiple Wallet ?

@gavinandresen

This comment has been minimized.

Contributor

gavinandresen commented Oct 21, 2013

Rebase needed.

@Tranz5

This comment has been minimized.

Tranz5 commented Nov 13, 2013

This is awesome work. I hope it makes it way to BTC soon. I've been playing with it for a bit now. Here are the issues I am working through.

  1. Export doesn't work. ( Tried making BitCoinGui exportAction public and referencing it in WalletView::gotoHistoryPage but that only gives me default wallet) Still working on different methods to fix this.
  2. The RPC commands loadwallet and unloadwallet do not reflect in gui. I think the gui needs a connect to rpc commands.
  3. When clicking on the transaction on the right side of the gui, the transaction button doesn't get focus.
  4. The gui can't create a new wallet. using loadwallet then unloadwallet rcp, then load button in gui works, but..

So far these are the only issues I have found.

Thanks again for the hard work. I'll share what I can when as I work through these changes. I am still new and trying to catchup as quick as possible. Any hints are appreciated it.

Happy Bitcoining!

@Tranz5

This comment has been minimized.

Tranz5 commented Dec 1, 2013

I also found that sign message didn't work with other wallets.

I have found a solution to all of these.

I can do a pull request to this version, if this actually has a chance to be part of btc. I don't have time to rebase though.

Thanks!

@bananas2

This comment has been minimized.

bananas2 commented Mar 24, 2014

Is it already implemented?

@laanwj

This comment has been minimized.

Member

laanwj commented Mar 24, 2014

No, the pull request was not kept up to date. I needs a lot of rebasing to apply to 0.9.x, which Gavin already noted 5 months ago.

@laanwj

This comment has been minimized.

Member

laanwj commented Apr 3, 2014

I'm going to close this. It has diverged too much from the current code base and no one seems to be interested in rebasing it.

In case anyone ever wants to have a shot at implementing multi-wallet I'll refer to to these code changes, as in principle they are good but they just arrived at the wrong time and were not kept in sync long enough to be tested properly and merged.

@laanwj laanwj closed this Apr 3, 2014

@laanwj laanwj referenced this pull request Apr 25, 2014

Closed

Multiple wallet #4093

@6coind

This comment has been minimized.

6coind commented Dec 19, 2015

Noone can re-base this ? Amazing that this did not become the standard

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment