Compact filters balance example

[RCasatta]

Alekos and I tested compact filters with bitcoin core 0.21 rc2, where compact filters are enabled at the p2p layer, it doesn't work at the moment because of rust-bitcoin/rust-bitcoin#529,

Maybe we will need this example to quickly test once things are deployed, however, it should probably be integrated into the repl to be merged

[afilini]

I think it makes sense to add compact_filters to the repl, but compared to what we did with esplora where we just enabled it by default I would keep the conditional compilation and only enable it if the compact_filters feature is enabled. librocksdb takes too long to build unfortunately.

[eupn]

@RCasatta rust-bitcoin/rust-bitcoin#529 is merged

[afilini]

Once #190 is in we can merge this one as well

[RCasatta]

I've still got errors with local bitcoin node on testnet:

2021-02-10T11:01:36Z peer 2673 requested unsupported block filter type: 0

even if I have blockfilterindex=1 in bitcoin.conf

https://github.com/bitcoin/bitcoin/blob/3931732191fbf21ee5a70eae73b44342bd91ab7b/src/net_processing.cpp#L2274

[RCasatta]

bitcoin node requires also peerblockfilters=1 to serve filters via p2p

[RCasatta]

Everything works fine now, about a minute to compute the balance, result balance is the same with electrum backend

[2021-02-10T11:36:15Z INFO  compact_filters_balance] start                                                                                                                                                                                                                      
...
[2021-02-10T11:36:54Z INFO  bdk::blockchain::compact_filters] Synced headers to height: 1935173
...
[2021-02-10T11:37:25Z INFO  compact_filters_balance] balance: 840000                                                                                                                                                                                                            

[notmandatory]

FYI I created a little program to search all the public testnet seed nodes and I found two that are supporting compact filters:

159.69.36.23 | version: 70016, services: ServiceFlags(NETWORK|WITNESS|COMPACT_FILTERS|NETWORK_LIMITED)
66.183.0.205 | version: 70016, services: ServiceFlags(NETWORK|WITNESS|COMPACT_FILTERS|NETWORK_LIMITED)

I then tested your CF balance example with 159.69.36.23 instead of localhost, from an empty wallet-filters db, and it worked great:

[2021-02-11T00:15:59Z INFO  compact_filters_balance] start
[2021-02-11T00:16:01Z INFO  compact_filters_balance] done CompactFiltersBlockchain { peers: [Peer { writer: Mutex { data: TcpStream { addr: 10.5.0.2:52818, peer: 159.69.36.23:18333, fd: 4 } }, responses: RwLock { data: {"version": (Mutex { data: [] }, Condvar { .. }), "verack": (Mutex { data: [] }, Condvar { .. }), "addr": (Mutex { data: [Addr([(1613002560, Address {services: ServiceFlags(NETWORK|WITNESS|COMPACT_FILTERS|NETWORK_LIMITED), address: 159.69.36.23, port: 18333})])] }, Condvar { .. })} }, reader_thread: JoinHandle { .. }, connected: RwLock { data: true }, mempool: Mempool(RwLock { data: InnerMempool { txs: {}, wtxids: {} } }), version: VersionMessage { version: 70016, services: ServiceFlags(1097), timestamp: 1613002560, receiver: Address {services: ServiceFlags(NONE), address: 45.155.40.84, port: 52818}, sender: Address {services: ServiceFlags(NETWORK|WITNESS|COMPACT_FILTERS|NETWORK_LIMITED), address: 0.0.0.0, port: 0}, nonce: 2207004403804454408, user_agent: "/Satoshi:0.21.0/", start_height: 1935331, relay: true }, network: Testnet },...], headers: ChainStore<Full> { cf_name: "default", min_height: 0, network: Testnet, headers_height: Ok(0), tip_hash: Ok(Some(000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943)) }, skip_blocks: Some(500000) }
[2021-02-11T00:16:01Z INFO  bdk::wallet] Derivation of 100 addresses from 0 took 20 ms
[2021-02-11T00:33:17Z INFO  bdk::blockchain::compact_filters] Applying snapshot with work: 0x000000000000000000000000000000000000000000000344f9d8e9f0de04c3ce
[2021-02-11T00:33:30Z INFO  bdk::blockchain::compact_filters] Synced headers to height: 1935334
[2021-02-11T00:47:46Z WARN  bdk::blockchain::compact_filters] Peer has BLOOM disabled, we can't ask for the mempool
...
[2021-02-11T00:47:49Z INFO  bdk::blockchain::compact_filters] Setting external index to 100
[2021-02-11T00:47:49Z INFO  bdk::blockchain::compact_filters] Dropping blocks until 1935234
[2021-02-11T00:47:49Z INFO  compact_filters_balance] balance: 840000

Time to full sync: 00:31:50

[notmandatory]

Hi @wpaulino, do you have any data from Neutrino about how long a single node testnet sync takes? Or maybe other compact block filter related performance tests we should look at? The bdk implementation is brand new so just trying to figure out how close it is to reasonable performance. Thanks!

[wpaulino]

@notmandatory haven't done any tests recently, but we were doing 5 min mainnet syncs with checkpoints and 2-3 peers last year on an iPhone. Testnet would take around 10 mins.

[afilini]

With checkpoints meaning that you would skip parts of the blockchain? Our time is for syncing the headers first and then downloading all the cf headers, cf themselves, processing them, downloading blocks, processing those, etc..

The wallet we are using to test doesn't really have that many UTXOs, so I don't think downloading/processing blocks is the bottleneck here. Mostly just syncing the headers and filters

[afilini]

Also, @notmandatory: did you compile in release or debug mode? Because especially for the "processing" part of compact filters I'd expect we could gain some time by using release mode, if you haven't done so already.. probably nothing major but it might help

[wpaulino]

With checkpoints meaning that you would skip parts of the blockchain? Our time is for syncing the headers first and then downloading all the cf headers, cf themselves, processing them, downloading blocks, processing those, etc.

Our time is just for syncing the block and filter header chains, no blocks were downloaded/processed. We use checkpoints to determine how big each batch of headers we request is, but we don't skip any.

[notmandatory]

I did an updated test (two runs) with RUST_LOG=info cargo run --release --features=compact_filters --example compact_filters_balance and this time using peer 66.183.0.205:18333 (because 159.69.36.23 isn't responding). The results are:

Run 1: Time to full sync: 00:07:57
Run 2: Time to full sync: 00:07:21

Not a perfect comparison since I had to use a different peer node, but still a good indication that with a release build it's possible to get much faster sync times! 🎉

Run 1

[2021-02-16T19:16:07Z INFO  compact_filters_balance] start
[2021-02-16T19:16:08Z INFO  compact_filters_balance] done CompactFiltersBlockchain { peers: [Peer { writer: Mutex { data: TcpStream { addr: 192.168.4.27:37066, peer: 66.183.0.205:18333, fd: 4 } }, responses: RwLock { data: {"verack": (Mutex { data: [] }, Condvar { .. }), "addr": (Mutex { data: [Addr([(1613502874, Address {services: ServiceFlags(NETWORK|WITNESS|COMPACT_FILTERS|NETWORK_LIMITED), address: 66.183.0.205, port: 18333})])] }, Condvar { .. }), "version": (Mutex { data: [] }, Condvar { .. })} }, reader_thread: JoinHandle { .. }, connected: RwLock { data: true }, mempool: Mempool(RwLock { data: InnerMempool { txs: {}, wtxids: {} } }), version: VersionMessage { version: 70016, services: ServiceFlags(1097), timestamp: 1613502874, receiver: Address {services: ServiceFlags(NONE), address: xxx.xxx.xxx.xxx, port: 37066}, sender: Address {services: ServiceFlags(NETWORK|WITNESS|COMPACT_FILTERS|NETWORK_LIMITED), address: 0.0.0.0, port: 0}, nonce: 8998089578838773924, user_agent: "/Satoshi:0.21.0/", start_height: 1936343, relay: true }, network: Testnet }, ..., headers: ChainStore<Full> { cf_name: "default", min_height: 0, network: Testnet, headers_height: Ok(0), tip_hash: Ok(Some(000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943)) }, skip_blocks: Some(500000) }
[2021-02-16T19:16:08Z INFO  bdk::wallet] Derivation of 100 addresses from 0 took 7 ms
[2021-02-16T19:20:40Z INFO  bdk::blockchain::compact_filters] Applying snapshot with work: 0x000000000000000000000000000000000000000000000365d739fe6244153c85
[2021-02-16T19:20:42Z INFO  bdk::blockchain::compact_filters] Synced headers to height: 1936344
[2021-02-16T19:24:03Z WARN  bdk::blockchain::compact_filters] Peer has BLOOM disabled, we can't ask for the mempool
[2021-02-16T19:24:04Z INFO  bdk::blockchain::compact_filters] Saving tx e0d1fb37ff165412f6f211de77a3107b506d0cae5367a0cc92fdfcc48306ba93
...
[2021-02-16T19:24:04Z INFO  bdk::blockchain::compact_filters] Saving tx 40eeecbd82a3b85d1d7600aed0cfb28c3858966641e427ec8398f423093602b4
[2021-02-16T19:24:04Z INFO  bdk::blockchain::compact_filters] Setting external index to 100
[2021-02-16T19:24:04Z INFO  bdk::blockchain::compact_filters] Dropping blocks until 1936244
[2021-02-16T19:24:04Z INFO  compact_filters_balance] balance: 1985561

Run 2

[2021-02-17T00:01:49Z INFO  compact_filters_balance] start
[2021-02-17T00:01:50Z INFO  compact_filters_balance] done CompactFiltersBlockchain { peers: [Peer { writer: Mutex { data: TcpStream { addr: 192.168.4.27:42154, peer: 66.183.0.205:18333, fd: 4 } }, responses: RwLock { data: {"verack": (Mutex { data: [] }, Condvar { .. }), "addr": (Mutex { data: [Addr([(1613520015, Address {services: ServiceFlags(NETWORK|WITNESS|COMPACT_FILTERS|NETWORK_LIMITED), address: 66.183.0.205, port: 18333})])] }, Condvar { .. }), "version": (Mutex { data: [] }, Condvar { .. })} }, reader_thread: JoinHandle { .. }, connected: RwLock { data: true }, mempool: Mempool(RwLock { data: InnerMempool { txs: {}, wtxids: {} } }), version: VersionMessage { version: 70016, services: ServiceFlags(1097), timestamp: 1613520015, receiver: Address {services: ServiceFlags(NONE), address: xxx.xxx.xxx.xxx, port: 42154}, sender: Address {services: ServiceFlags(NETWORK|WITNESS|COMPACT_FILTERS|NETWORK_LIMITED), address: 0.0.0.0, port: 0}, nonce: 7414547177834796079, user_agent: "/Satoshi:0.21.0/", start_height: 1936380, relay: true }, network: Testnet }, ...], headers: ChainStore<Full> { cf_name: "default", min_height: 0, network: Testnet, headers_height: Ok(0), tip_hash: Ok(Some(000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943)) }, skip_blocks: Some(500000) }
[2021-02-17T00:01:50Z INFO  bdk::wallet] Derivation of 100 addresses from 0 took 5 ms
[2021-02-17T00:05:41Z INFO  bdk::blockchain::compact_filters] Applying snapshot with work: 0x0000000000000000000000000000000000000000000003670829fb4ecc2a89fc
[2021-02-17T00:05:43Z INFO  bdk::blockchain::compact_filters] Synced headers to height: 1936380
[2021-02-17T00:09:10Z WARN  bdk::blockchain::compact_filters] Peer has BLOOM disabled, we can't ask for the mempool
[2021-02-17T00:09:10Z INFO  bdk::blockchain::compact_filters] Saving tx e0d1fb37ff165412f6f211de77a3107b506d0cae5367a0cc92fdfcc48306ba93
...
[2021-02-17T00:09:10Z INFO  bdk::blockchain::compact_filters] Saving tx 40eeecbd82a3b85d1d7600aed0cfb28c3858966641e427ec8398f423093602b4
[2021-02-17T00:09:10Z INFO  bdk::blockchain::compact_filters] Setting external index to 100
[2021-02-17T00:09:10Z INFO  bdk::blockchain::compact_filters] Dropping blocks until 1936280
[2021-02-17T00:09:10Z INFO  compact_filters_balance] balance: 1985561