Bitcoin core RPC backend #348

RCasatta · 2021-05-17T15:23:18Z

updated version of #22

closes #79 and #86

~~needs~~ rebased on top of #347 for common blockchain tests

Description

Implements Blockchain trait for a backend using an RPC node

Notes to the reviewers

Note we are obliged to use importmulti because bitcoin core doesn't support all descriptor https://bitcoindevkit.org/descriptors/#compatibility-matrix

use bitcoind crate as dev-dependency, requires env var pointing to a bitcoind executable, see for example https://github.com/RCasatta/bitcoind/blob/master/.github/workflows/test.yml

Checklists

All Submissions:

I've signed all my commits
I followed the contribution guidelines
I ran cargo fmt and cargo clippy before committing

New Features:

I've added tests for the new feature
I've added docs for the new feature
I've updated CHANGELOG.md

jb55 · 2021-05-18T17:24:20Z

oh hi, deja vu! I need this for a project, trying it now...

jb55 · 2021-05-18T18:55:55Z

src/blockchain/rpc.rs

+    ImportMultiRequestScriptPubkey, ImportMultiRescanSince,
+};
+use bitcoincore_rpc::jsonrpc::serde_json::Value;
+use bitcoincore_rpc::{Auth, Client, RpcApi};


I think we should re-export bitcoincore_rpc in src/lib.rs ? RpcConfig depends on Auth so you can't construct it otherwise.

diff --git a/src/lib.rs b/src/lib.rs index f5323b1..477f023 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -216,6 +216,9 @@ extern crate bdk_macros; #[cfg(feature = "compact_filters")] extern crate lazy_static; +#[cfg(feature = "rpc")] +pub extern crate bitcoincore_rpc; + #[cfg(feature = "electrum")] pub extern crate electrum_client;

sure, adding...

Before final merge maybe we can assess if only Auth is needed we may want to import only that struct

jb55 · 2021-05-18T22:59:34Z

src/blockchain/rpc.rs

+
+        //TODO if current_height == node_synced should check only the mempool
+
+        //TODO split the interval in chunk so that we can give progress update


instead of start/stop params which involves state, we could just periodically call getwalletinfo for the progress.

yes, that is better

reconsidered again, see #348 (comment)

jb55 · 2021-05-19T17:21:45Z

src/blockchain/rpc.rs

+
+        //TODO if it's a big rescan, spawn a thread that checks getwalletinfo and update progress
+        self.client
+            .rescan_blockchain(Some(node_synced as usize), Some(current_height as usize))?;


one thing I noticed is that if this call is ever interrupted during a long rescan from 0, set_node_synced_height is never called, and then the next time sync happens it will rescan from 0 again. not sure what the best way to handle this is.

Yeah, right, maybe it's better to return to the initially proposed logic of doing the rescan in chunks, for example, every 10_000 blocks, this would solve both the progress (without a separated thread for the getwalletinfo calss) and make the restart from crash less bad

jb55 · 2021-05-20T00:20:05Z

src/blockchain/rpc.rs

+}
+
+impl RpcBlockchain {
+    fn get_node_synced_height(&self) -> Result<u32, Error> {


what do you think about making these methods public? I was able to implement a "quick sync" for new wallets by setting node_synced_height to the current tip.

Maybe it should be a config parameter?

in compact_filters is passed at creation time https://docs.rs/bdk/0.7.0/bdk/blockchain/compact_filters/struct.CompactFiltersBlockchain.html#method.new

jb55 · 2021-05-20T13:48:25Z

Yeah, right, maybe it's better to return to the initially proposed logic of doing the rescan in chunks, for example, every 10_000 blocks, this would solve both the progress (without a separated thread for the `getwalletinfo` calss) and make the restart from crash less bad

yup I came to the same conclusion after thinking about it some more 👍

jb55 · 2021-05-20T13:50:31Z

Maybe it should be a config parameter? in compact_filters is passed at creation time https://docs.rs/bdk/0.7.0/bdk/blockchain/compact_filters/struct.CompactFiltersBlockchain.html#method.new

either way works for me! having a similar interface to the other backends sounds like a good thing.

RCasatta · 2021-05-21T14:14:56Z

rebased on top of #347

src/blockchain/rpc.rs

jb55 · 2021-05-25T04:40:13Z

we should export this under the blockchain module like the others:

diff --git a/src/blockchain/mod.rs b/src/blockchain/mod.rs
index 95069ec..b9f5e3c 100644
--- a/src/blockchain/mod.rs
+++ b/src/blockchain/mod.rs
@@ -43,6 +43,11 @@ pub use self::electrum::ElectrumBlockchain;
 #[cfg(feature = "electrum")]
 pub use self::electrum::ElectrumBlockchainConfig;
 
+#[cfg(feature = "rpc")]
+pub use self::rpc::RpcBlockchain;
+#[cfg(feature = "rpc")]
+pub use self::rpc::RpcConfig;
+
 #[cfg(feature = "esplora")]
 #[cfg_attr(docsrs, doc(cfg(feature = "esplora")))]
 pub mod esplora;

RCasatta · 2021-05-26T13:31:28Z

Rebased on master and squashed

afilini · 2021-06-01T07:22:34Z

src/blockchain/mod.rs

+#[cfg(feature = "rpc")]
+pub mod rpc;
+#[cfg(feature = "rpc")]
+pub use self::rpc::RpcBlockchain;
+#[cfg(feature = "rpc")]
+pub use self::rpc::RpcConfig;


The new RPC variant should be added to src/blockchain/any.rs as well

Yes, right, but I think we need to defer this (maybe following PR) until rust-bitcoin/rust-bitcoincore-rpc#181 is merged and released (initial work for reference d13b925)

src/blockchain/rpc.rs

src/testutils/blockchain_tests.rs

src/blockchain/rpc.rs

afilini · 2021-06-01T07:59:48Z

src/blockchain/rpc.rs

+        for tx_result in list_txs.iter().filter(|t| {
+            // list_txs returns all conflicting tx we want to
+            // filter out replaced tx => unconfirmed and not in the mempool
+            t.info.confirmations > 0 || self.client.get_mempool_entry(&t.info.txid).is_ok()


Maybe here we should differentiate between the tx not being in the mempool and other random errors like networking issues

If that's not too hard

Not sure we can discriminate between random errors and not being in the mempool...

src/blockchain/rpc.rs

jb55 suggested changes May 18, 2021

View reviewed changes

jb55 reviewed May 18, 2021

View reviewed changes

RCasatta force-pushed the rpc branch from 6f5b027 to 5d8174a Compare May 19, 2021 07:47

jb55 reviewed May 19, 2021

View reviewed changes

jb55 reviewed May 20, 2021

View reviewed changes

RCasatta force-pushed the rpc branch from f45006d to dc96b57 Compare May 21, 2021 14:14

RCasatta commented May 21, 2021

View reviewed changes

src/blockchain/rpc.rs Outdated Show resolved Hide resolved

RCasatta force-pushed the rpc branch from 036fd42 to 41cb8d4 Compare May 26, 2021 13:31

RCasatta force-pushed the rpc branch 6 times, most recently from 5dd8833 to c111e5c Compare May 31, 2021 11:33

RCasatta marked this pull request as ready for review May 31, 2021 11:36

RCasatta force-pushed the rpc branch from 73c6614 to 2731eb3 Compare May 31, 2021 11:38

afilini reviewed Jun 1, 2021

View reviewed changes

RCasatta added 7 commits June 3, 2021 10:55

Implements RPC Backend

bfef2e3

update changelog with rpc feature

0b96965

match also on signet

dffb753

rename struct to CallResult

9b7ed08

change feature to execute sync from rpc to test-rpc

e1b037a

correctly initialize UTXO keychain kind

8185119

always ask node for tx no matter capabilities

ab98283

RCasatta added 2 commits June 3, 2021 11:06

use storage address instead of satoshi's

ae5aa06

update bitcoind dep

ab54a17

RCasatta force-pushed the rpc branch from c35de83 to ab54a17 Compare June 3, 2021 09:27

RCasatta added 2 commits June 3, 2021 15:26

move scan in setup

1639984

add details to TODO, format doc example

ba2e304

afilini approved these changes Jun 11, 2021

View reviewed changes

afilini merged commit 1825411 into bitcoindevkit:master Jun 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bitcoin core RPC backend #348

Bitcoin core RPC backend #348

RCasatta commented May 17, 2021 •

edited

jb55 commented May 18, 2021

jb55 May 18, 2021

RCasatta May 19, 2021 •

edited

jb55 May 18, 2021

RCasatta May 19, 2021

RCasatta May 31, 2021

jb55 May 19, 2021

RCasatta May 20, 2021

jb55 May 20, 2021

RCasatta May 20, 2021

jb55 commented May 20, 2021 via email

jb55 commented May 20, 2021 via email

RCasatta commented May 21, 2021

jb55 commented May 25, 2021

RCasatta commented May 26, 2021

afilini Jun 1, 2021

RCasatta Jun 1, 2021

afilini Jun 1, 2021

afilini Jun 1, 2021

RCasatta Jun 2, 2021


		//TODO if current_height == node_synced should check only the mempool

		//TODO split the interval in chunk so that we can give progress update

Bitcoin core RPC backend #348

Bitcoin core RPC backend #348

Conversation

RCasatta commented May 17, 2021 • edited

Description

Notes to the reviewers

Checklists

All Submissions:

New Features:

jb55 commented May 18, 2021

Choose a reason for hiding this comment

RCasatta May 19, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jb55 commented May 20, 2021 via email

jb55 commented May 20, 2021 via email

RCasatta commented May 21, 2021

jb55 commented May 25, 2021

RCasatta commented May 26, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RCasatta commented May 17, 2021 •

edited

RCasatta May 19, 2021 •

edited