Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feasibility for replacing Bitcoinj with Bitcoin Core SPV #1062

Closed
ManfredKarrer opened this issue Dec 4, 2017 · 15 comments
Closed

Feasibility for replacing Bitcoinj with Bitcoin Core SPV #1062

ManfredKarrer opened this issue Dec 4, 2017 · 15 comments
Assignees
@ManfredKarrer
Labels
Epic re:Bitcoin

Comments

@ManfredKarrer
Copy link
Member

Bitcoin Core SPV (Jonas Schnelli's branch) should have similar resource requirements like BitcoinJ. Investigate if that is true (downloaded data at first start up) and if it is feasible investigate best integration option. We could ship it as binary like we do with Tor and communicate via RPC. Investigate if RPC is best option and if it is fast enough (it is not very fast).
@ManfredKarrer
Copy link
Member Author

Neutrino might be an interesting option as well:
https://github.com/lightninglabs/neutrino

@chirhonul
Copy link
Member

Yes, I'd suggest we look into Neutrino (BIP157/BIP158)-based setups instead of bloom filters (BIP37). BIP37 is generally seen among core devs as fundamentally incompatible with privacy, and also imposes disproportional cost on full nodes (DoS vector). The main drawback of the GCS-based light node protocol is that CPU cost can be higher for the client, which could make things slow for a mobile device, but as Bisq is a desktop that should not be an issue for us.

I'd be happy to contriute here, once I understand more how the bisq -> bitcoinJ integration is built.

@ManfredKarrer
Copy link
Member Author

@chirhonul Are you familiar with the Jonas Schnelli's approach?
bitcoin/bitcoin#9483

We had a past tech session where i gave an overview about the Wallet/BitcoinJ part:
https://www.youtube.com/watch?v=JcySykHuWIQ&index=4&list=PLFH5SztL5cYOtcg64PntHlbtLoiO3HAjB

@chirhonul
Copy link
Member

Yes @ManfredKarrer, I have been following the bitcoin/bitcoin#9483 PR.

I will take a look at the tech session recording, thanks!

@chirhonul
Copy link
Member

I've watched the video. It had some interesting details that I now understand better, thank you for linking it!

Here is one project that may be interesting, since you mentioned in the video that integrating Bitcoin Core with Java would be hard / impossible, at least for Android. The ABCore project achieves this, using the Android NDK (https://developer.android.com/ndk/) to have a thin Java layer wrapping the Bitcoin Core C++ codebase inside an Android .apk. It's possible to install it and download a (pruned) blockchain on an Android device. I have tried it, it takes a very long time to sync the chain but does work:

Since Android Studio is based on IntelliJ and uses Gradle, it might not be so hard to use the same setup to compile and package up C++ code like Bitcoin Core and bundle it inside the Bisq binary, if the RPC latency would be significant.

I understand the desire to continue working with the BitcoinJ fork if possible and gradually improve the code in our fork or upstream (or a new project forming a permanent fork, if necessary). I am not familiar with BitcoinJ at all basically so that would take some warm-up period for me personally, but I do have a fairly good understanding of BIP44, BIP37, BIP157/BIP158 etc. I would be up for either working on improving our BitcoinJ fork, or planning for a migration towards client-side filtering, as well as thinking about what JVM-based lightweight client may be useful to the wider ecosystem beyond Bisq. We can maybe discuss details on short-term tasks I can start with as well as broader strategy in other channels.

@ManfredKarrer
Copy link
Member Author

Ah thanks for the info regarding the ABCore project . Have not heard about that.
Though as a full Bisq version for Mobile is not really planned we have lower requirements than most other projects.

We might be able to integrate Bitcoin Core the same way like we integrate the Tor binary (Bisq starts the shipped Tor binary as new process and talks via a TPC interface to it, for Bitcoin Core we could use RPC and/or ZeroMQ). I think that would give the good performance of a native app and we are always in sync with latest reference client.

If that approach does not work (if the SPV model of Bitcoin Core is not feasible regarding resource behaviour) we should maybe get in touch with Samurai Wallet and Mycelium developers. They share a lot our mindset regarding privacy and at least Samurai Wallet has already implemented SegWit.

I am in touch with a BitcoinJ developer who has signaled to be interested to work on Bisq after he has completed a current project (ends around August).

@oscarguindzberg
Copy link
Contributor

I have investigated this subject in detail.

Super short summary: I found no good replacement for bitcoinj that we could use today. bip157/bip158 is where the industry is going, so it is worth keeping an eye on it.
Contributing bip157/bip158 support to bitcoinj would be an interesting option to evaluate if someone decides to invest time on this.

In general light wallet design faces a dilemma: Privacy (prevent others knowing what addresses are mine) and Trust (don’t trust third parties) vs. Low resource consumption (storage, cpu, bandwith).

An ideal bitcoinj replacement for bisq should have these features:

  • Light - does not requires huge amounts of bandwidth/cpu/disk (ie no bitcoin core).
  • Hierarchical Deterministic/BIP32 support (good example: bitcoinj).
  • Tor Support.
  • Stable, production tested and actively supported.
  • Censorship resistant and Privacy - don’t allow the server owner or MITM to know what txs are mine. Worse example: Coinbase. Bad examples: known-to-be-broken bloom filters/bip37/bitcoinj, electrum, copay. Good examples: bip157/bip158. Perfect case: bitcoin core.
  • High-level API: e.g. bitcoinj's wallet class.
  • Mobile support is not a requirement (bisq is just desktop)

Resources to understand bitcoin privacy conscious wallet start-of-the-art

Some ideas to make a light wallet:

  • Download block headers until wallet creation date and full blocks from wallet creation date.
  • UTXO set commitments (root hash of UTXO set in the block header / coinbase tx).
    • You have to trust previous blocks follow consensus rules.
    • You can do full validation from that point on, if you download the rest of the blocks.
  • Checkpoints: Have hardcoded checkpoints in the source code. Don’t download block headers before the checkpoints.
  • Encrypt communication between the wallet and the server (BIP 151, electrum, etc).
  • Pruning: remove old blocks (or data in general) as you already validated it and it is past the finality threshold (ie 100+ blocks).
  • Client-side filtering: (BIP 157/158)

Now, the projects I evaluated...

Bitcoinj

  • Benefits
    • Already implemented in Bisq.
    • Tor Support.
    • Stable, production tested.
    • Unconfirmed txs friendly.
    • High-level API: e.g. wallet class.
  • problems
    • Bad performance for arbitrators. Once you get lots of keys and txs in your wallet things are very slow (probably similar performance to not using bloom filters at all).
    • Unmaintained: no release for 2,5 years
    • No segwit support.
    • Bloom filters problems

Jonas schinelli’s “hybrid full block / SPV mode” on bitcoin core

  • Modify bitcoin core to download block headers until wallet creation date and full blocks from wallet creation date.
  • https://bitcoincore.org/en/2017/03/31//prioritized-block-download-using-hybrid-spv/
  • Downloading the “old” blocks is optional: Wallet could do it to avoid following a longer but consensus-breaking chain.
  • Idea: add pruning support to require less storage.
  • Benefits
    • Full node privacy.
    • It’s bitcoin core, so it avoids potential alternative client bugs.
    • Recover from existing wallet works faster and the user can start using the wallet (even if she is not sure the shown balance/tx belong to blocks with correct PoW but invalid consensus).
    • Also makes the process of new wallet creation much faster since just headers and no blocks will be downloaded
  • Problems
    • Required resources are similar or equal to full bitcoin node.
    • Unfinished/Abandoned implementation.
  • Unmerged PRs

Neutrino / bip157 / bip158

  • https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
  • https://github.com/bitcoin/bips/blob/master/bip-0158.mediawiki
  • How it works
    • Bitcoin full nodes provide a filter for each block.
    • Filter is created deterministically by the full node based on the block. The filter is the same filter for every wallet.
    • Light wallets check client-side whether the filter shows the block might contain a relavant tx for them, if it does, they download the full block.
  • Built to be used by lightning network nodes
  • Benefits
    • Similar privacy to a full node: Peers nor MITM can find out what addressed belong to the light wallet.
    • Much less bandwith/storage/cpu required compared to a full node.
  • Problems
    • Far from production ready technology.
    • No solution for unconfirmed txs (light wallets should download every broadcasted tx or don’t be aware of unconfirmed txs at all)
    • More bandwith required compared to bloom filters or electrum:
      • Filter size is 20k per block, much bigger than once in a while bloom filter updates.
      • It has to download full blocks if a relevant tx might be in the block, no SPV proof usage.
    • To investigate: Are there problems with the false/positive rate?
      • Is one-size-fits-all false/positive rate a problem?
      • Clients will end up downloading full blocks that don’t contain relevant txs to the wallet. The more addresses the wallet has, the more unnecessary blocks it will download. Is this a non-problem for lightning network nodes that have just a few keys to monitor (lots of keys added and discarded frequently) but a real problem for normal wallets that have to monitor a lot of derived-long-time-ago-but-still-valid keys.
    • How to make sure the filter is valid?
      • There is no filter commitment yet. A soft fork to add a filter hash to the coinbase tx / block header is very far away (maybe in 2 years).
      • Currently there is a (weak?) mechanism to find out if a peer is providing fraudulent filters (ie connecting to several peers, requesting information from all of them and eventually downloading full blocks)
  • Server side implementations:
  • Client side implementations
    • Neutrino
      • https://github.com/lightninglabs/neutrino
      • Reference client implementation for bip157 / bip158.
      • golang
      • lower level api compared to bitcoinj
        • no class similar to bitcoinj wallet (no coin selection, method to get balances, key derivation)
      • alpha stage
        • docs say “experimental” / “not recommended to use it with real money”
        • almost no user documentation
      • Created for the needs of the LND client
    • Wasabi includes a bip157 / bip158 client implementation (written in c#)
    • JVM
      • Acinq/eclair told they plan to port neutrino to scala/kotlin
      • Muun is slowly building a bitcoin lib in java and they plan to add bip157/158 support.
      • A Bisq contributor might add bip157 / bip158 client support to bitcoinj

Bitcoin scala implementations

Samurai wallet

Wasabi Wallet

Wallets using centralized servers

  • Most prominent examples: Copay, Mycelium, electrum.
  • Benefits
    • Almost instant new wallet creation and seed recovery.
    • Lowest bandwidth/storage/cpu alternative available.
    • Stable, production tested and actively supported.
    • Encrypted Client-server communication.
  • Problems
    • The server knows exactly my addresses.
    • The server can not inform users of incoming txs.
    • Users should trust the company server or servers run by random people they don’t know.
    • Personal Electrum Server could, in theory, be run by users, but it requires a local bitcoin core which kills the almost instant on and low resource consumption benefits.

@oscarguindzberg
Copy link
Contributor

Btw, I had a quick look and discarded alternatives listed in https://bitcoin.org/en/choose-your-wallet :

  • Bitcoin knots: it is a full node
  • Green address: centralized
  • mSIGNA: requires a full node
  • ArcBit: centralized
  • Armory: python. requires a full node
  • Bither: java. Uses bloom filters. Uses a custom library written in java (https://github.com/bither/bitherj ), similar to bitcoinj, but not based on bitcoinj. Seems dated.

@ManfredKarrer
Copy link
Member Author

Thanks @oscarguindzberg for the summary!
A small note: Bisq uses it's own tor infrastructure so Tor support is not really needed (we don't use BitcoinJ's now abandoned Tor support which was also not supporting hidden services).

@Christewart
Copy link

Christewart commented Dec 12, 2018 •
edited

@oscarguindzberg Great list. Me and my company maintain bitcoin-s and are planning on building a lite client with https://github.com/bitcoin-s/bitcoin-s-core. Currently we have rpc and core protocol functionality along with signing logic. Things that still need to be built is a stand alone wallet and p2p networking services. The goal is to allow this library to be used for a lite client for lightning network nodes, but I think it should satisfy bisqs requirements too.

@lontivero
Copy link

Just for the record: NBitcoin supports all these features and more. Client side compact filters and its p2p protocol (bip158 and bip157), all existing standard scripts, bech32 addresses, partially signed transactions (reviewing now), rpc client, rest client, schnorr signature (early impl), etc.
There was a time when bitcoinj was ahead of NBitcoin but currently NBitcoin is far far away, features are implemented few months after core (sometimes nbitcoin implements bips before core, ex: bip158).

Sent from my Moto G(4) using FastHub

@ManfredKarrer
Copy link
Member Author

@lontivero Ah thanks. A pity it's not in Java....

@oscarguindzberg oscarguindzberg mentioned this issue Dec 31, 2018
Closed
@oscarguindzberg oscarguindzberg mentioned this issue May 9, 2019
Open
@schildbach
Copy link

FYI: Bitcoin Core plans to switch off server-side filtering by default. The change has already been merged and they plan to release it with the next major version. Node operators can still enable filtering manually.

The relevant thread is on the bitcoin-dev mailing list. Topic: "Bitcoin Core to disable Bloom-based Filtering by default"

@ManfredKarrer
Copy link
Member Author

@schildbach Thanks for the info!

I expected that will happen some day and I can understand the reasoning behind it. Luckily we use our provided nodes so we an handle it, but shows that BIP 37 SPV mode reaches its final days....

@ManfredKarrer
Copy link
Member Author

I agree to that statement:
bitcoin/bitcoin#16152 (comment)

I think as public network will become infeasible and provided BTC nodes the only working option (beside local node or private node) moving to a federated electrum server model might be better in the mid term. Will also solve many performance issues and the difficulty to find experienced BitcoinJ devs.

@chimp1984 chimp1984 mentioned this issue Nov 13, 2019
Closed
@oscarguindzberg oscarguindzberg mentioned this issue May 23, 2020
Closed
2 tasks
@chimp1984 chimp1984 mentioned this issue Sep 29, 2020
Closed
@chimp1984 chimp1984 mentioned this issue Apr 29, 2021
Closed
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ManfredKarrer ManfredKarrer

Labels
Epic re:Bitcoin
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@lontivero @ripcurlx @schildbach @oscarguindzberg @ManfredKarrer @Christewart @chirhonul