Introduce block-by-block API to `bdk::Wallet` and add RPC wallet example #1172

vladimirfomene · 2023-10-12T10:29:11Z

Description

Introduce block-by-block API for bdk::Wallet. A wallet_rpc example is added to demonstrate syncing bdk::Wallet with the bdk_bitcoind_rpc chain-source crate.

The API of bdk_bitcoind_rpc::Emitter is changed so the receiver knows how to connect to the block emitted.

Notes to the reviewers

Changelog notice

Added

Wallet methods to apply full blocks (apply_block and apply_block_connected_to) and a method to apply a batch of unconfirmed transactions (apply_unconfirmed_txs).
CheckPoint::from_block_ids convenience method.
LocalChain methods to apply a block header (apply_header and apply_header_connected_to).
Test to show that LocalChain can apply updates that are shorter than original. This will happen during reorgs if we sync wallet with bdk_bitcoind_rpc::Emitter.

Fixed

InsertTxError now implements std::error::Error.

All Submissions:

I've signed all my commits
I followed the contribution guidelines
I ran cargo fmt and cargo clippy before committing

New Features:

I've added tests for the new feature
I've added docs for the new feature

crates/bdk/src/wallet/mod.rs

example-crates/wallet_rpc/src/main.rs

vladimirfomene · 2023-10-23T07:55:56Z

@evanlinjin apology for the delay on this one. I'm doing all necessary changes today.

evanlinjin · 2023-11-06T20:20:39Z

Is the intention to have the user supply their own descriptor? Or are we hardcoding the descriptor?

vladimirfomene · 2023-11-09T08:00:09Z

I think we should hardcode the descriptor as we do in other wallet example

ValuedMammal

I did successfully run the regtest example, and after modifying wallet_rpc/src/main.rs to correctly parse the arg vector, I ran the example on both testnet and signet. I had to play with setting an appropriate fallback height to find the wallet txs, and on one occasion the program sat hanging while applying blocks, but I didn't investigate why. I wonder if this example might also be adapted to run on regtest, but that can be saved for future work

crates/bdk/src/wallet/mod.rs

example-crates/wallet_rpc/src/main.rs

vladimirfomene · 2023-11-10T13:04:46Z

@ValuedMammal, I have added a note for other reviewers to check if the program hangs while applying blocks.

example-crates/example_bitcoind_rpc_polling/README.md

adds Wallet methods `set_lookahead_for_all`, `apply_block_relevant`, `batch_insert_relevant_unconfirmed`

danielabrozzoni

Looks good to me!

crates/bdk/src/wallet/mod.rs

example-crates/example_bitcoind_rpc_polling/README.md

danielabrozzoni

ACK d97875b

example-crates/example_bitcoind_rpc_polling/README.md

darosior

I would suggest the wallet use a default lookahead limit other than 0. I expect it would be quite surprising to the user of the high-level Wallet interface to see that he has to explicitly call set_lookahead on the wallet for it to be able to notice a utxo received on next_deriv_index + 1.

Doesn't need to be in this PR, but wanted to raise it here since it's tightly related to the functionality introduced.

I think 1_000 would be a good default. This is what the Bitcoin Core wallet uses.

EDIT: this can actually be tackled in parallel of this PR. I've done so here: #1229.

crates/bdk/src/wallet/mod.rs

LLFourn

This LGTM but I'd like to ask whether the changes to the file store actually result in a significant performance improvement? I think the issue is a design bug in the bincode API which is fixed in the v2 release which returns the amount of data read from the reader. Maybe we can just wait for that instead of making this more complicated.

evanlinjin · 2024-01-12T23:32:09Z

@LLFourn wrapping the file in BufReader is a significant improvement.

For the counting reader, we still need to back track on any type of error. To backtrack, we still need the position before the read. It's either we seek to get the position, or we store the position.

Side note, maybe BufReader caches seeks?

evanlinjin · 2024-01-13T10:17:57Z

@LLFourn I moved bdk_file_store changes into a separate PR: #1270

ValuedMammal

I think overall this is good. It's nice to see everything come together in wallet_rpc in an elegant fashion, can confirm the examples work as intended.

In b486ecc 'feat(wallet): introduce block-by-block api', the commit description could be updated (still referring to methods process_block, etc).

We should perhaps include more links to example crates in the workspace README (wallet_rpc, example_esplora, example_bitcoind_rpc).

ValuedMammal · 2024-01-14T19:31:21Z

crates/bdk/src/wallet/mod.rs

+
+/// An error that may occur when applying a block to [`Wallet`].
+#[derive(Debug)]
+pub enum ApplyBlockError {


Did you intend to use this for either apply_block or apply_block_connected_to?

ValuedMammal · 2024-01-14T19:36:13Z

crates/bdk/src/wallet/mod.rs

+    /// This method takes in an iterator of `(tx, last_seen)` where `last_seen` is the timestamp of
+    /// when the transaction was last seen in the mempool. This is used for conflict resolution
+    /// when there is conflicting unconfirmed transactions. The transaction with the later
+    /// `last_seen` is prioritied.


typo: prioritied

ValuedMammal · 2024-01-14T19:43:09Z

crates/chain/src/local_chain.rs

+                }
+                ApplyHeaderError::CannotConnect(err) => err,
+            })
+    }


Can this function return ApplyHeaderError, or is mapping the err serving some greater purpose?

It should probably be called ApplyHeaderConnectedTo error.

ValuedMammal · 2024-01-14T20:11:07Z

example-crates/example_bitcoind_rpc_polling/README.md

feel free to ignore: I found this example a bit difficult to follow. It might be better suited for a website tutorial, but if we decide to keep it, I made a few edits in terms of style ValuedMammal/bdk@0caeb79

This looks great, can you do a PR?

This looks great, can you do a PR?

Onto this branch or on master?

LLFourn · 2024-01-15T06:18:45Z

crates/chain/src/local_chain.rs

+    /// The `header` will be transformed into checkpoints - one for the current block and one for
+    /// the previous block. Note that a genesis header will be transformed into only one checkpoint
+    /// (as there are no previous blocks). The checkpoints will be applied to the chain via
+    /// [`apply_update`].


The documentation doesn't tell us what the method is for and is written in the passive voice which makes it harder to read. It should start by saying that it's going to insert the block hash into the chain along with the previous block if it's not already there. The documentation shouldn't mention apply_update -- the reader doesn't care and if we change this then the documentation will be broken.

LLFourn

ACK db58b8a

These are convenience methods to transform a header into checkpoints to update the `LocalChain` with. Tests are included.

* methods `process_block` and `process_unconfirmed_txs` are added * amend stage method docs Co-authored-by: Vladimir Fomene <vladimirfomene@gmail.com> Co-authored-by: 志宇 <hello@evanlinjin.me>

Previously, emissions are purely blocks + the block height. This means emitted blocks can only connect to previous-adjacent blocks. Hence, sync must start from genesis and include every block.

Previously, `apply_block_relevant` used `batch_insert_relevant` which allows inserting non-topologically-ordered transactions. However, transactions from blocks are always ordered, so we can avoid looping through block transactions twice (as done in `batch_insert_relevant`). Additionally, `apply_block_relevant` now takes in a reference to a `Block` instead of consuming the `Block`. This makes sense as typically very few of the transactions in the block are inserted.

Co-authored-by: Vladimir Fomene <vladimirfomene@gmail.com> Co-authored-by: 志宇 <hello@evanlinjin.me>

LLFourn · 2024-01-15T23:31:07Z

self-ACK: a4f28c0

I added one commit to improve the documentation of apply_header_connected_to

stevenroose · 2024-01-16T16:00:06Z

I would like to use this :)

evanlinjin · 2024-01-17T05:47:33Z

I would like to use this :)

Me too! cc. @kcalvinalvin

evanlinjin

ACK a4f28c0

51bd01b fix(file_store): recover file offset after read (志宇) 66dc34e refactor(file_store): Use BufReader but simplify (LLFourn) c871764 test(file_store): `last_write_is_short` (志宇) a3aa8b6 feat(file_store)!: optimize `EntryIter` by reducing syscalls (志宇) Pull request description: ### Description `EntryIter` performance is improved by reducing syscalls. The underlying file reader is wrapped with `BufReader` (to reduce calls to `read` and `seek`). Two new tests are introduced. One ensures correct behavior when the last changeset write is too short. The other ensures the next write position is correct after a short read. ### Notes to the reviewers This is extracted from #1172 as suggested by #1172 (review). ### Changelog notice Changed * `EntryIter` performance is improved by reducing syscalls. ### Checklists #### All Submissions: * [x] I've signed all my commits * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md) * [x] I ran `cargo fmt` and `cargo clippy` before committing #### New Features: * [x] I've added tests for the new feature * [x] I've added docs for the new feature ACKs for top commit: LLFourn: ACK 51bd01b Tree-SHA512: 9c25f9f2032cb2d551f3fe4ac62b856ceeb69a388f037b34674af366c55629a2eaa2b90b1ae4fbd425415ea8d02f44493a6c643b4b1a57f4507e87aa7ade3736

vladimirfomene force-pushed the implement_rpc_wallet branch from 2dc9575 to ef8a2ab Compare October 12, 2023 10:39

vladimirfomene self-assigned this Oct 12, 2023

evanlinjin requested changes Oct 16, 2023

View reviewed changes

crates/bdk/src/wallet/mod.rs Outdated Show resolved Hide resolved

crates/bdk/src/wallet/mod.rs Outdated Show resolved Hide resolved

crates/bdk/src/wallet/mod.rs Outdated Show resolved Hide resolved

example-crates/wallet_rpc/src/main.rs Outdated Show resolved Hide resolved

evanlinjin mentioned this pull request Oct 17, 2023

Add RPC Wallet Example #1106

Closed

8 tasks

nondiremanuel added this to the 1.0.0-alpha.3 milestone Oct 22, 2023

nondiremanuel mentioned this pull request Oct 22, 2023

W43 Lib Team Call bitcoindevkit/.github#83

Closed

33 tasks

vladimirfomene force-pushed the implement_rpc_wallet branch 3 times, most recently from ff5ecca to 45ddedc Compare October 24, 2023 15:04

nondiremanuel mentioned this pull request Nov 5, 2023

W45 Lib Team Call bitcoindevkit/.github#84

Closed

38 tasks

vladimirfomene force-pushed the implement_rpc_wallet branch from 45ddedc to 3b6b8e2 Compare November 6, 2023 08:47

ValuedMammal reviewed Nov 9, 2023

View reviewed changes

crates/bdk/src/wallet/mod.rs Show resolved Hide resolved

example-crates/wallet_rpc/src/main.rs Outdated Show resolved Hide resolved

example-crates/wallet_rpc/src/main.rs Outdated Show resolved Hide resolved

vladimirfomene force-pushed the implement_rpc_wallet branch from 3b6b8e2 to 7a24c1a Compare November 10, 2023 12:56

vladimirfomene force-pushed the implement_rpc_wallet branch 2 times, most recently from a0e45bc to d97875b Compare November 10, 2023 13:38

evanlinjin reviewed Nov 16, 2023

View reviewed changes

example-crates/example_bitcoind_rpc_polling/README.md Show resolved Hide resolved

ValuedMammal added a commit to ValuedMammal/bdk that referenced this pull request Nov 19, 2023

Note: Drop this when bitcoindevkit#1172 is merged

6cabc5f

adds Wallet methods `set_lookahead_for_all`, `apply_block_relevant`, `batch_insert_relevant_unconfirmed`

This was referenced Nov 19, 2023

W47 Lib Team Call bitcoindevkit/.github#86

Closed

Need a method to set the index lookahead for Wallet #1166

Closed

danielabrozzoni reviewed Nov 21, 2023

View reviewed changes

crates/bdk/src/wallet/mod.rs Outdated Show resolved Hide resolved

example-crates/example_bitcoind_rpc_polling/README.md Show resolved Hide resolved

danielabrozzoni approved these changes Nov 22, 2023

View reviewed changes

example-crates/example_bitcoind_rpc_polling/README.md Show resolved Hide resolved

vladimirfomene force-pushed the implement_rpc_wallet branch 2 times, most recently from 4fbfabb to fad6f94 Compare November 23, 2023 07:40

darosior reviewed Nov 27, 2023

View reviewed changes

darosior mentioned this pull request Nov 27, 2023

Use a universal lookahead value for KeychainTxOutIndex and have a reasonable default #1229

Merged

4 tasks

darosior reviewed Nov 27, 2023

View reviewed changes

crates/bdk/src/wallet/mod.rs Outdated Show resolved Hide resolved

evanlinjin requested review from darosior and ValuedMammal January 11, 2024 08:39

LLFourn reviewed Jan 12, 2024

View reviewed changes

evanlinjin mentioned this pull request Jan 13, 2024

Improve performance of bdk_file_store::EntryIter #1270

Merged

5 tasks

evanlinjin force-pushed the implement_rpc_wallet branch from f9d07fa to db58b8a Compare January 13, 2024 10:15

evanlinjin requested a review from LLFourn January 13, 2024 10:17

ValuedMammal reviewed Jan 14, 2024

View reviewed changes

evanlinjin mentioned this pull request Jan 15, 2024

Avoid using BTreeMap::append #1274

Merged

4 tasks

LLFourn reviewed Jan 15, 2024

View reviewed changes

LLFourn approved these changes Jan 15, 2024

View reviewed changes

evanlinjin and others added 7 commits January 16, 2024 00:23

feat(chain): add CheckPoint::from_block_ids convenience method

2b61a12

feat(chain): add apply_header.. methods to LocalChain

d3e5095

These are convenience methods to transform a header into checkpoints to update the `LocalChain` with. Tests are included.

feat(wallet): introduce block-by-block api

9467cad

* methods `process_block` and `process_unconfirmed_txs` are added * amend stage method docs Co-authored-by: Vladimir Fomene <vladimirfomene@gmail.com> Co-authored-by: 志宇 <hello@evanlinjin.me>

test(chain): LocalChain test for update that is shorter than original

8f2d4d9

feat(bitcoind_rpc)!: emissions include checkpoint and connected_to data

e0512ac

Previously, emissions are purely blocks + the block height. This means emitted blocks can only connect to previous-adjacent blocks. Hence, sync must start from genesis and include every block.

feat(example): add RPC wallet example

8ec65f0

Co-authored-by: Vladimir Fomene <vladimirfomene@gmail.com> Co-authored-by: 志宇 <hello@evanlinjin.me>

evanlinjin force-pushed the implement_rpc_wallet branch from db58b8a to 8ec65f0 Compare January 15, 2024 16:28

chore: improve LocalChain::apply_header_connected_to doc

a4f28c0

evanlinjin approved these changes Jan 19, 2024

View reviewed changes

evanlinjin merged commit 25653d7 into bitcoindevkit:master Jan 19, 2024
12 checks passed

notmandatory mentioned this pull request Jan 30, 2024

Release 1.0.0-alpha.5 #1303

Closed

23 tasks

This was referenced Mar 6, 2024

Implement PrunedBlockchain backend: RPC on pruned node #530

Closed

Update bdk_bitcoind_rpc module to work with a pruned bitcoind node #1371

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce block-by-block API to `bdk::Wallet` and add RPC wallet example #1172

Introduce block-by-block API to `bdk::Wallet` and add RPC wallet example #1172

vladimirfomene commented Oct 12, 2023 •

edited by evanlinjin

vladimirfomene commented Oct 23, 2023

evanlinjin commented Nov 6, 2023

vladimirfomene commented Nov 9, 2023

ValuedMammal left a comment

vladimirfomene commented Nov 10, 2023 •

edited

danielabrozzoni left a comment

danielabrozzoni left a comment

darosior left a comment •

edited

LLFourn left a comment •

edited

evanlinjin commented Jan 12, 2024 •

edited

evanlinjin commented Jan 13, 2024

ValuedMammal left a comment

ValuedMammal Jan 14, 2024

ValuedMammal Jan 14, 2024

ValuedMammal Jan 14, 2024

LLFourn Jan 15, 2024

ValuedMammal Jan 14, 2024

evanlinjin Jan 15, 2024

ValuedMammal Jan 16, 2024

LLFourn Jan 15, 2024 •

edited

LLFourn left a comment

LLFourn commented Jan 15, 2024

stevenroose commented Jan 16, 2024

evanlinjin commented Jan 17, 2024

evanlinjin left a comment

Introduce block-by-block API to bdk::Wallet and add RPC wallet example #1172

Introduce block-by-block API to bdk::Wallet and add RPC wallet example #1172

Conversation

vladimirfomene commented Oct 12, 2023 • edited by evanlinjin

Description

Notes to the reviewers

Changelog notice

All Submissions:

New Features:

vladimirfomene commented Oct 23, 2023

evanlinjin commented Nov 6, 2023

vladimirfomene commented Nov 9, 2023

ValuedMammal left a comment

Choose a reason for hiding this comment

vladimirfomene commented Nov 10, 2023 • edited

danielabrozzoni left a comment

Choose a reason for hiding this comment

danielabrozzoni left a comment

Choose a reason for hiding this comment

darosior left a comment • edited

Choose a reason for hiding this comment

LLFourn left a comment • edited

Choose a reason for hiding this comment

evanlinjin commented Jan 12, 2024 • edited

evanlinjin commented Jan 13, 2024

ValuedMammal left a comment

Choose a reason for hiding this comment

ValuedMammal Jan 14, 2024

Choose a reason for hiding this comment

ValuedMammal Jan 14, 2024

Choose a reason for hiding this comment

ValuedMammal Jan 14, 2024

Choose a reason for hiding this comment

LLFourn Jan 15, 2024

Choose a reason for hiding this comment

ValuedMammal Jan 14, 2024

Choose a reason for hiding this comment

evanlinjin Jan 15, 2024

Choose a reason for hiding this comment

ValuedMammal Jan 16, 2024

Choose a reason for hiding this comment

LLFourn Jan 15, 2024 • edited

Choose a reason for hiding this comment

LLFourn left a comment

Choose a reason for hiding this comment

LLFourn commented Jan 15, 2024

stevenroose commented Jan 16, 2024

evanlinjin commented Jan 17, 2024

evanlinjin left a comment

Choose a reason for hiding this comment

Introduce block-by-block API to `bdk::Wallet` and add RPC wallet example #1172

Introduce block-by-block API to `bdk::Wallet` and add RPC wallet example #1172

vladimirfomene commented Oct 12, 2023 •

edited by evanlinjin

vladimirfomene commented Nov 10, 2023 •

edited

darosior left a comment •

edited

LLFourn left a comment •

edited

evanlinjin commented Jan 12, 2024 •

edited

LLFourn Jan 15, 2024 •

edited