[Wallet] improve/replace wallet

[jonasschnelli]

I open this issue to have a space for discussions and a status of current work going on.

There are plans to mostly rewrite the current wallet implementation and – speaking long term – completely decouple the wallet from bitcoin-core and move it to a separate repository. This might lower down the amount of pull requests and issues, etc. in the bitcoin-core repository.

Planed transition/features:

• Decouple wallet from bitcoin core, remove wallets #ifdef ENABLE_WALLET and implement a startup command to run bitcoin core with a/the new wallet or the legacy wallet (-addmodule=wallet or -addmodule=legacywallet, or -addmodule=wallet,internalminer) Introduce basic modularization #5752, refactor: move BDB (bitdb / db.h) interaction from init.cpp to wallet.cpp #5758, [Modularization] add possibility to extend or overwrite RPC calls #5744
• Keep the legacywallet as untouched as possible
• Add a new BDB free wallet with a internal append only data format (logdb) (this could also be the starting point of a new wallet module). Support a migration tool. [Wallet] replace BDB with internal append only (logdb) backend #5686
• Remove accounting system (new wallet only) or transform it into labels Remove bolt-on accounting system #3816
• Add support for Bip32/HDWallets
• Add SPV mode
• Add multiwallet support

Status

• Currently the legacywallet decoupling is the main focus. See Introduce basic modularization #5752, refactor: move BDB (bitdb / db.h) interaction from init.cpp to wallet.cpp #5758, [Modularization] add possibility to extend or overwrite RPC calls #5744, [Move Only] Move wallet related sources to src/wallet/ #5745
• The planed internal append only file format has been proven to work in [Wallet] replace BDB with internal append only (logdb) backend #5686
• HTTP/RPC Server: libevent-based http server #5677 (which is related somehow)

Related Discussions

#3882 Split off wallet to seperate repository
#3440 Post-0.9 modularization

---

If someone is interested in contributing, it would mainly help to review and test the PRs listed above. Of course delivering implementation would also be great, just make sure to speak up in a comment and make others aware of what your are working on.

As usual, the bottleneck will be testing and reviewing. Especially the unspectacular changes which not directly lead to a feature like bip32 or spv at first sight.

Conceptional ideas are also very much welcomed.

[luke-jr]

It makes far more sense (easier, etc) to start the new wallet code in a separate repository as a library, rather than start it here to be moved out later...

[jonasschnelli]

@luke-jr Yes and no. Because the core needs modularization first, it makes sense IMO to start in this repository. Otherwise i'm pretty sure it will end up in one of these bitcoin forks with no attention.
Currently it's to early to split off.

[gmaxwell]

I think moving it to a separate repository would be a mistake for the following reasons:

(1) A separate repository would get less review and analysis. We have objective evidence from libsecp256k1 which is absolutely essential to the system, both in the form of lower traffic in its repo and some comments only happening when we go to merge an update to Bitcoin core.

(2) A separate repository will create more difficulty for people fetching the project from git to use it and a steeper learning curve for new developers. The 'btcd' project, as I understand, has recently switched from many repositories to escape these difficulties.

(3) A separate repository increases the risk of version drift where users suffer unexpected behavior because they are using never tested before mixed versions. (Though this is less of an issue the more SPV the new wallet is)

(4) There is also less opportunity for code sharing due to the need to sync across both projects.

So I think this should not be a goal in and of itself. If the new wallet ends up mostly being "orthogonally maintained" in practice, then it might make sense to split it after all... but we should let that practice drive the procedure there, not the other way around. :)

There is a long list of wallet features hiding someplace that I found interesting that I can probably share with you. I'll look for it.

[jtimon]

Wouldn't subrepositories be a better solution when/if we separate them?

[luke-jr]

(1) is probably a symptom of Bitcoin Core being excessively centralised right now, and only going to change when we get it more modular. (2) is easily solved with an umbrella repository with submodules. (4) may be a concern/delay, but may help us understand better what code needs to be in which library.

[laanwj]

Although I don't disagree with any of @gmaxwell's arguments, we should not ignore the organizational side. The scope of this project is too broad. If the wallet is to grow beyond a 'reference wallet'. To have a successful wallet project it needs to be maintained by people focused on developing a wallet, and also have specific goal. E.g. end-user wallet application or scalable, multi-user enterprise wallet? Or a library that can be used to develop both kinds? All are valid goals, persued on their own.

Currently it's everything in between, a Rorschach blot of features. There's no clear direction. Which makes sense as this is the general 'bitcoin' project. Different people all expect something else from it. It's good enough for many things but not excellent at any. People then think this is an inherent limitation of Bitcoin instead of just a heavily bottlenecked dev team.

Also: As I've mentioned many times, having issues about the UI, wallet, and global consensus in one repository is very distracting. I think this repository should eventually focus on maintaining the P2P node. That doesn't mean 'drop the wallet now!'. Maintaining an example wallet is OK. But by definition it will be very generic.

Having said that (sorry for hijacking this thread, but some things need to be said), I'm ok with developing the "newwallet" as part of this repository for the shorter term, and with the changes outlined in @jonasschnelli's post. But in the medium to longer term I believe the current way of working is a burden.

[maflcko]

See also #18916

[maflcko]

@jonasschnelli Can this be closed? Some of the points are no longer relevant (accounts have been removed, #ifdef ENABLE_WALLET have been removed, ...)

I think we should continue discussion in #18916 or open a new issue if anything else needs to be done.

[jarolrod]

@MarcoFalke @jonasschnelli I think this can be closed since we've introduced an sqlite wallet and other changes.