[WIP] Explore wallet addresses using an embedded bwt electrum server

[shesek]

Add support for exploring wallet addresses derived from a set of descriptors/xpubs provided by the user, by running an embedded personal bwt electrum server and pointing btc-rpc-explorer to it.

Example usage: --address-api bwt --descriptor 'wpkh(xpub69..oP/0/*)' --xpub xpub66..zQ

This is based on the libbwt C FFI and the libbwt-nodejs package , which are still work-in-progress and not officially released. So this is not quite ready yet, I'm just posting this to gather feedback and see if there's interest. -- which has since been officially released.

Some notes:

• I did not add the libbwt npm package as a dependency. Instead, users are instructed to install it separately if they wish to use it.
• When bwt is used as the backend, the "trust note" messages are not shown because the information is as reliable as anything else that comes from the full node.
• I added instructions for the bwt backend to the README. But since the README is rather minimal, it might not be the best place for it?
• I prefixed all of the bwt options with BWT_/bwt- to keep them separated, but this makes them kinda long. Let me know if you prefer something else.

[shesek]

Another possibility that I did not explore in this PR is using the bwt HTTP API, which would allow to display some additional information, such as:

• A list of tracked wallet descriptors/xpubs/addresses
• A list of recent transactions across all wallet addresses
• Highlight inputs/outputs relating to the wallet when viewing transactions
• Display the descriptor associated with wallet addresses
• Show the net effect the transaction has on the wallet's balance

With this, btc-rpc-explorer could be used not only as a block explorer, but also as a watch-only wallet explorer, which is pretty neat. :)

[Kixunil]

What's the advantage of this over just using bitcoin cores watch-only wallet?

[shesek]

@Kixunil This is using a Bitcoin Core watch-only wallet behind the scenes. But:

• Bitcoin Core does not support querying for the transaction history of a specified address, which is the primary thing that btc-rpc-explorer needs to do with addresses. bwt maintains an address history index that allows making these lookups.

• Bitcoin Core requires manually importing addresses and maintaining the gap limit as more addresses get used. With bwt this is done automatically, you just configure the xpubs/descriptors and the rest is taken care of.

• This PR reuses the existing code for the Electrum backend, without requiring a new backend implementation for working with addresses through Bitcoin Core.

So basically, yes, you can do everything directly with Bitcoin Core and without bwt, it'll just require quite a bit more work.

[Kixunil]

Bitcoin Core does not support querying for the transaction history of a specified address

That's surprising, since the GUI can show history, but never tried with watch-only wallet.

Anyway, it makes sense after what you explained.

I'm wondering, isn't a limitation of BWT that you have to specify the xpub on the command line?
Also, does this support cross compilation? (Rust itself does, but I've seen node projects that are unable to handle it correctly, that's why I'm asking.)

[shesek]

That's surprising, since the GUI can show history, but never tried with watch-only wallet.

Bitcoin Core allows querying for the history of the entire wallet, but not of specific addresses.

I'm wondering, isn't a limitation of BWT that you have to specify the xpub on the command line?

Do you mean having to specify an xpub as the limitation, or having to do this on the command line?

Also, does this support cross compilation?

bwt does support cross-compilation. The nodejs-bwt-daemon package uses pre-built libbwt library files that are available for linux, osx, windows, arm32v7 and arm64v8.

Build instructions are available here.

[Kixunil]

Having to do it on command line. While having to specify xpub can also lead to degraded experience, it can be a reasonable trade-off for people who don't want to store a few more GB of data.

The nodejs-bwt-daemon package uses pre-built libbwt library files that are available for linux-x64, osx-x64, windows-x64, arm32v7 and arm64v8.

Oh, that's nice and as that doc says, it's reproducible. The reason I'm asking is I'm trying to understand what kind of overhead merging of this would add to me packaging btc-rpc-explorer since I also cross-compile for arm64 and node stuff was quite difficult to get working (node build systems don't seem to be of high quality when compared to other languages).

[shesek]

Having to do it on command line.

I can add other configuration methods, how would you suggest to do this?

(note that everything is already configurable using environment variables, which could be used with the .env file.)

what kind of overhead merging of this would add to me packaging btc-rpc-explorer

What are you packaging it for?

and node stuff was quite difficult to get working (node build systems don't seem to be of high quality when compared to other languages)

nodejs-bwt-daemon doesn't actually use any of the nodejs build systems, it has a simple postinstall script that grabs the libbwt library file from github releases (and verify its hash against the SHA256SUMS file that ships with the npm package).

If you're packaging btc-rpc-explorer alongside all of its node_modules dependencies, you could simply put the library file for the target platform inside node_modules/bwt-daemon and everything should work.

[Kixunil]

I can add other configuration methods, how would you suggest to do this?

Definitely via UI. When the feature is enabled, user should have the ability to paste the xpub. There should also be a list of dynamically added xpubs, that user can preview or modify. Would be nice if the user could also label the xpub. (The experience would be roughly similar to what github does with SSH keys.)

What are you packaging it for?

Currently x86_64 and (lower priority) arm64 Debian 10 (Buster) via my Deb repository.
Thinking about it a bit, maybe if you added a very tiny standalone UI, ideally with token authentication and multi-user support then I could package BWT itself. I've purposefully avoided doing so (same for EPS) because not having to manually edit config files is hard requirement. (ports etc. can be automated, xpubs can not)

nodejs-bwt-daemon doesn't actually use any of the nodejs build systems, it has a simple postinstall script that grabs the libbwt library file from github releases (and verify its hash against the SHA256SUMS file that ships with the npm package).

That's not a bad option, since it has all architectures I currently care about.

From the script: process.env.npm_config_arch - is this target architecture? That means the architecture on which the compiled output will run, not the architecture npm is currently running on?

if (!process.env.BWT_NO_DOWNLOAD) { - this is handy anyway :)

[shesek]

Definitely via UI.

This is definitely something that I'm planning to add to bwt eventually, but it might take me a little while to get there. One complication is that bwt is currently entirely stateless (apart from the state stored by the bitcoin core wallet), similarly to electrum-personal-server and to btc-rpc-explorer, so this would require adding a persistent storage backend first.

While not ideal, I think that a configuration file could still be good enough for the initial integration. Users that are running a full node are typically more tech-savvy and used to command line and configuration files, so it would provide value to at least some of the users.

From the script: process.env.npm_config_arch - is this target architecture?

That is correct, it is.

nodejs-bwt-daemon doesn't actually use any of the nodejs build systems

This wasn't quit accurate. While nodejs-bwt-daemon itself doesn't, it does depend on node-ffi-napi, which is itself a native module that uses node-gyp-build.

[Kixunil]

Users that are running a full node are typically more tech-savvy and used to command line and configuration files

I strive to do better than that. I want everyone to be able to run a node as easily as any other app. It can be a stopgap, sure, but not the final solution. This is the exact reason EPS is not packaged in my repository and will not be until this changes. Being stateless may sound nice and flashy to programmers (like "blockchain" ;)) but it's entirely useless if the consequence is 99% of people can't use an application.

The storage can be as simple as a CSV file (label, xpub) that gets rewritten (written to a temp file first, then fsync, then move over) when a modification is requested by a user. There can be a few additional authenticated methods for manipulating the list. Then creating UI should be quite easy.

Thinking about it, the whole thing should be authenticated because otherwise if someone gets access to BWT, it's possible to start requesting each address from the chain to see which belong to the user. Unfortunately Electrum doesn't support any kind of authentication, so network namespace isolation might be the only possibility.

[shesek]

I strive to do better than that. I want everyone to be able to run a node as easily as any other app. It can be a stopgap, sure, but not the final solution

Agreed.

Thinking about it, the whole thing should be authenticated

Agreed too. But as you said, the Electrum protocol doesn't really provide a proper way to do this.

Note, however, that in the setup in this PR, the electrum server is only binding on localhost, so it would have to be an attacker with local access.

[Kixunil]

so it would have to be an attacker with local access.

Which is exactly something I like to prevent as well because local isolation is useful too (reducing damage). My repository isolates everything that makes sense for instance.

[shesek]

I updated to the latest libbwt-nodejs v0.2.1 release and resolved the merge conflict.

Would love to hear your thoughts @janoside :-)

[janoside]

@shesek After responding to #279 I see I owe you some feedback here too...sorry. The basic feedback is the same: this is great and I look forward to getting this merged.

I took a stab at working on this one recently, but ran into some trouble getting libbwt installed and hadn't budgeted enough time to investigate, so had to punt.

(FYI - Always excited when I see one of your PRs come in, but I also know that they're going to be an investment because you're so prolific! Thanks for all your contributions and I'll keep trying to keep up.)

[shesek]

Sounds fantastic, thank you!

Do you remember what trouble you ran into with libbwt?