nakamoto as Compact Block Filter Blockchain #751

rajarshimaitra · 2022-09-13T06:42:15Z

Description

An implementation of nakamoto as a BDK Blockchain to serve as an alter BIP157 syncing mechanism.

Commit 1 : adds a new module src/blockchain/compact_filters/nakamoto.rs which defines a CbfNode struct which holds the namakoto-client handlers. Which the implements Blockchain traits. 222a2b4

Commit 2 : Integrates the CbfNode backend with the bdk's blockchain test framework.

Notes to Reviewers

The current CI tests won't work because nakamoto doesn't support our MSRV. Though you can run the modified tests in your local with toolchain 1.63 or higher.

Few of the tests are intermittently failing and needs further investigation. But overall this PR is ready for review of to be used for testing..

Thanks for all the help and support from @cloudhead and others who pitched in for various fixes in upstream.

Checklists

All Submissions:

I've signed all my commits
I followed the contribution guidelines
I ran cargo fmt and cargo clippy before committing

cloudhead

Looks good to me.

cloudhead · 2022-10-09T07:59:47Z

Cargo.toml

+nakamoto-net =  { git = "https://github.com/cloudhead/nakamoto.git", branch = "master", optional = true }
+nakamoto-net-poll =  { git = "https://github.com/cloudhead/nakamoto.git", branch = "master", optional = true }
+nakamoto-p2p = { git = "https://github.com/cloudhead/nakamoto.git", branch = "master", optional = true }
+nakamoto-common = { git = "https://github.com/cloudhead/nakamoto.git", branch = "master", optional = true }


You should be able to just import the nakamoto crate. It exposes these externally.

Thanks for the tip.. I missed it.. Yes that's much more cleaner..

cloudhead · 2022-10-09T08:05:09Z

src/blockchain/compact_filters/nakamoto.rs

+                    transactions,
+                } => {
+                    let timestamp = header.time;
+                    for tx in transactions {


Note that all transactions of the matching block are included here, so you have to make sure to discard the ones you aren't interested in.

We are actually handling that in the add_tx itself.. So any unrelated transaction will not get added in the db.. But ya the will be iterated over here but with not much extra over head as some basic if let statements in add_tx won't trigger for them. Maybe there's a better way to organize this, but for start it made sense to consolidate all transaction conflicts and addition logic in single place in add_tx..

Here we are just checking if the transaction hasn't been already processed by us..

cloudhead · 2022-10-09T08:09:51Z

src/blockchain/compact_filters/nakamoto.rs

+                        .collect::<HashSet<_>>();
+
+                    for txid in db_txs {
+                        unconfirm_tx(database, &txid)?;


Note that nakamoto should already be emitting a TxStatus::Reverted for each tx in a reverted block, so you shouldn't have to handle it here.

Hmm okay makes sense.. We still handle reverted cases in TxStatus::Reverted.. I am commenting this section out for now..

cloudhead · 2022-10-09T08:16:04Z

src/blockchain/compact_filters/nakamoto.rs

+                        debug!("Sync complete at : {}", height);
+                        self.last_sync_height.set(height as u32);
+                        break;
+                    }


I assume you have another mechanism to restart the event loop when a new block header is received.

No, this is basically a hack to stop the sync loop for the reorg tests in regetst.. Ideally an invalidateblock call on the regtest should also update the tip height, and the sync should automatically stop at the new tip height, but that wasn't happening.. So instead this hack is to force stop the sync.. This is feature gated as test, and cannot be used in production..

Ah gotcha, ok.

cloudhead · 2022-10-09T08:19:16Z

src/blockchain/compact_filters/nakamoto.rs

+                        }
+                        TxStatus::Reverted => {
+                            debug!("Transaction reverted due to reorg: txid : {}", txid);
+                            let _ = unconfirm_tx(database, &txid)?;


FYI, nakamoto will re-announce reverted transactions. I don't think we fire another event for them other than this one (but we could).

I think this transaction reversion event should be sufficient to catch them..

rajarshimaitra · 2022-10-12T07:02:44Z

Thanks @cloudhead for the review and suggestion.. It seems the structure of the crate has changed in recent master.. So I pinned the rev to 14902d3d5cc9442cdda82419f0f24277d86b243b. I couldn't find any release tag in github, and the published v0.3.0 in crates.io doesn't seem to include some latest fixes.. Anyway, this works for now, until we get a more stable release of nakamoto out in crates.io..

rajarshimaitra · 2022-10-12T07:03:10Z

@thunderbiscuit this now includes AnyBlockhain and ConfigurableBlockchain updates..

thunderbiscuit

Hey Raj! So I'm trying to expose this in the bindings and it surfaces two small things: (1) I think the names for the other blockchain config structs are slightly different, (2) the fields in the config struct are currently private but I'd need access to them.

To get an idea for how we use the config structs (and how we wrap them up slightly, note the ElectrumBlockchainConfig to ElectrumConfig same for the Esplora) you can take a look at lines 98ish to 225ish of this source file.

src/blockchain/compact_filters/nakamoto.rs

rajarshimaitra · 2022-10-13T04:36:07Z

@thunderbiscuit updated as suggested..

rajarshimaitra · 2022-10-13T04:49:17Z

One more extra commit is added to disable to existing compact_filters logic, so we don't need to unnecessarily build rocksdb and other stuffs not used for nakamoto client..

thunderbiscuit · 2022-10-13T14:40:20Z

Working on the FFI integration here: bitcoindevkit/bdk-ffi#207
With this wallet: Devkit Wallet Nakamoto branch

This commit adds the nakamoto blockchain into the current blockchain test frameworks. Uses a lot of feature gate and scaffolding to handle special behaviors of CBF type backend.

Include CbfNode in AnyBlockchain enum Impl ConfigurableBlockchain for CbfNode

This commit comments out existing compact filters module so only nakamoto dependencies get compiled with compact_filters features.

danielabrozzoni · 2023-03-16T17:11:59Z

Hey, we are in the process of releasing BDK 1.0, which will under the hood work quite differently from the current BDK. For this reason, I'm closing all the PRs that don't really apply anymore. If you think this is a mistake, feel free to rebase on master and re-open!

rajarshimaitra mentioned this pull request Sep 14, 2022

Update rocksdb to 0.19 #724

Closed

9 tasks

rajarshimaitra force-pushed the cbf branch from a713f0f to d6d0320 Compare October 2, 2022 06:19

rajarshimaitra mentioned this pull request Oct 2, 2022

Consequtive sync is not working cloudhead/nakamoto#94

Closed

rajarshimaitra force-pushed the cbf branch 2 times, most recently from 9fe539c to a62a8d6 Compare October 8, 2022 17:33

rajarshimaitra marked this pull request as ready for review October 8, 2022 17:42

rajarshimaitra changed the title ~~[WIP] nakamoto as Compact Block Filter Blockchain~~ nakamoto as Compact Block Filter Blockchain Oct 8, 2022

cloudhead reviewed Oct 9, 2022

View reviewed changes

notmandatory assigned rajarshimaitra Oct 10, 2022

notmandatory added the enhancement New feature or request label Oct 10, 2022

notmandatory mentioned this pull request Oct 10, 2022

W41 BDK Library Team Call bitcoindevkit/.github#27

Closed

26 tasks

rajarshimaitra force-pushed the cbf branch from a62a8d6 to 5be8ef0 Compare October 12, 2022 06:59

rajarshimaitra force-pushed the cbf branch 2 times, most recently from ef6328e to d190bbb Compare October 12, 2022 07:14

rajarshimaitra marked this pull request as draft October 12, 2022 07:18

thunderbiscuit reviewed Oct 12, 2022

View reviewed changes

src/blockchain/compact_filters/nakamoto.rs Outdated Show resolved Hide resolved

src/blockchain/compact_filters/nakamoto.rs Outdated Show resolved Hide resolved

rajarshimaitra force-pushed the cbf branch from d190bbb to c6e246e Compare October 13, 2022 04:35

rajarshimaitra force-pushed the cbf branch from c6e246e to 0a14d45 Compare October 13, 2022 04:48

thunderbiscuit mentioned this pull request Oct 18, 2022

Add Compact Block Filters Blockchain Client bitcoindevkit/bdk-python#27

Closed

rajarshimaitra added 4 commits February 26, 2023 14:26

Add nakamoto blockchain

57112af

Add nakamoto into blockchain test

48b3757

This commit adds the nakamoto blockchain into the current blockchain test frameworks. Uses a lot of feature gate and scaffolding to handle special behaviors of CBF type backend.

Add nakamoto into extra infra

7e12db5

Include CbfNode in AnyBlockchain enum Impl ConfigurableBlockchain for CbfNode

Disable existing Compact Filters module

58c12e2

This commit comments out existing compact filters module so only nakamoto dependencies get compiled with compact_filters features.

rajarshimaitra force-pushed the cbf branch from 0a14d45 to 58c12e2 Compare February 26, 2023 09:00

nakamoto v0.4.0 update

7ef3786

Add initial peer list into config.

5e9603c

rajarshimaitra force-pushed the cbf branch from 944993a to 5e9603c Compare February 26, 2023 16:53

danielabrozzoni closed this Mar 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nakamoto as Compact Block Filter Blockchain #751

nakamoto as Compact Block Filter Blockchain #751

rajarshimaitra commented Sep 13, 2022 •

edited

Loading

cloudhead left a comment

cloudhead Oct 9, 2022

rajarshimaitra Oct 12, 2022

cloudhead Oct 9, 2022

rajarshimaitra Oct 12, 2022

cloudhead Oct 9, 2022

rajarshimaitra Oct 12, 2022

cloudhead Oct 9, 2022

rajarshimaitra Oct 12, 2022

cloudhead Oct 12, 2022

cloudhead Oct 9, 2022

rajarshimaitra Oct 12, 2022

rajarshimaitra commented Oct 12, 2022

rajarshimaitra commented Oct 12, 2022 •

edited

Loading

thunderbiscuit left a comment •

edited

Loading

rajarshimaitra commented Oct 13, 2022

rajarshimaitra commented Oct 13, 2022

thunderbiscuit commented Oct 13, 2022 •

edited

Loading

danielabrozzoni commented Mar 16, 2023

nakamoto as Compact Block Filter Blockchain #751

nakamoto as Compact Block Filter Blockchain #751

Conversation

rajarshimaitra commented Sep 13, 2022 • edited Loading

Description

Notes to Reviewers

Checklists

All Submissions:

cloudhead left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rajarshimaitra commented Oct 12, 2022

rajarshimaitra commented Oct 12, 2022 • edited Loading

thunderbiscuit left a comment • edited Loading

Choose a reason for hiding this comment

rajarshimaitra commented Oct 13, 2022

rajarshimaitra commented Oct 13, 2022

thunderbiscuit commented Oct 13, 2022 • edited Loading

danielabrozzoni commented Mar 16, 2023

rajarshimaitra commented Sep 13, 2022 •

edited

Loading

rajarshimaitra commented Oct 12, 2022 •

edited

Loading

thunderbiscuit left a comment •

edited

Loading

thunderbiscuit commented Oct 13, 2022 •

edited

Loading