fix(esplora): use saturating_add in update_tx_graph() #1110

notmandatory · 2023-09-04T05:32:34Z

Description

This fixes overflow error when calling update_tx_graph() from update_tx_graph_without_keychain().

Notes to the reviewers

You can reproduce the error by reverting 66a2bf5.

The tests could use some cleanup but get the job done for this PR.

Changelog notice

None

Checklists

All Submissions:

I've signed all my commits
I followed the contribution guidelines
I ran cargo fmt and cargo clippy before committing

Bugfixes:

This pull request breaks the existing API
I've added tests to reproduce the issue which are now passing
I'm linking the issue being fixed by this PR

LLFourn · 2023-09-04T06:06:48Z

I'm confused. How can this possibly overflow in practice?

notmandatory · 2023-09-04T06:41:39Z

I ran into this when I was testing sync with esplora but only checking unused spks, based on how the cli example works. I could be misunderstanding how that's supposed to be done, but I thought I could just give the esplora client my spks via update_tx_graph_without_keychain() and I'd get back any new tx. But since this passes a stop_gap of u32::Max it causes an overflow with any last_active_index > 0.

notmandatory · 2023-09-04T19:18:36Z

I was also able to repro this issue (using the original code) using the example-crates/example_esplora with these steps:

create a new address

 cargo run -- --network signet "wpkh(tprv8ZgxMBicQKsPdy6LMhUtFHAgpocR8GC6QmwMSFpZs7h6Eziw3SpThFfczTDh5rW2krkqffa11UpX3XkeTTB2FvzZKWXqPY54Y6Rq4AQ5R8L/84'/1'/0'/0/*)" "wpkh(tprv8ZgxMBicQKsPdy6LMhUtFHAgpocR8GC6QmwMSFpZs7h6Eziw3SpThFfczTDh5rW2krkqffa11UpX3XkeTTB2FvzZKWXqPY54Y6Rq4AQ5R8L/84'/1'/0'/1/*)" address new

send from faucet to new address

sync unused spks (don't think the parallel requests matter but that's how I did it)

cargo run -- --network signet "wpkh(tprv8ZgxMBicQKsPdy6LMhUtFHAgpocR8GC6QmwMSFpZs7h6Eziw3SpThFfczTDh5rW2krkqffa11UpX3XkeTTB2FvzZKWXqPY54Y6Rq4AQ5R8L/84'/1'/0'/0/*)" "wpkh(tprv8ZgxMBicQKsPdy6LMhUtFHAgpocR8GC6QmwMSFpZs7h6Eziw3SpThFfczTDh5rW2krkqffa11UpX3XkeTTB2FvzZKWXqPY54Y6Rq4AQ5R8L/84'/1'/0'/1/*)" sync --unused-spks --parallel-requests 2

LLFourn · 2023-09-05T22:45:46Z

crates/esplora/tests/blocking_ext.rs

+        1,
+    )?;
+
+    assert_eq!(graph_update.full_txs().count(), 2);


💡 test that the txids match those returned from send_to_address.

Fixed.

There should be a way to consolidate blocking and async tests and TestEnv (probably with macros as Alekos originally did), I'll create an issue.

LLFourn · 2023-09-05T22:49:32Z

crates/esplora/Cargo.toml

+# This feature is used to run `cargo check` in our CI targeting wasm. It's not recommended
+# for libraries to explicitly include the "getrandom/js" feature, so we only do it when
+# necessary for running our CI. See: https://docs.rs/getrandom/0.2.8/getrandom/#webassembly-support
+dev-getrandom-wasm = ["getrandom/js"]


why do we need this. Where do we need to use getrandom at all here?

It looks like getrandom is brought in via the electrsd dev-dependency I added. One improvement I can make is to only include getrandom as a dev-dependency since that's the only place it's needed.

# from esplora directory $ cargo tree -i getrandom getrandom v0.2.10 └── rand_core v0.6.4 ├── rand v0.8.5 │ └── secp256k1 v0.27.0 │ └── bitcoin v0.30.1 │ ├── bdk_chain v0.5.0 (/Users/steve/git/notmandatory/bdk/crates/chain) │ │ └── bdk_esplora v0.3.0 (/Users/steve/git/notmandatory/bdk/crates/esplora) │ ├── bitcoincore-rpc-json v0.17.0 │ │ └── bitcoincore-rpc v0.17.0 │ │ └── bitcoind v0.33.1 │ │ └── electrsd v0.25.0 │ │ [dev-dependencies] │ │ └── bdk_esplora v0.3.0 (/Users/steve/git/notmandatory/bdk/crates/esplora)

I think you don't need it at all since electrsd doesn't run in WASM anyway.

For some reason cargo check with the WASM target is still including getrandom and throwing the error. I'll see if I can find some way to force it not to.

yeah because it's compiling dev depends in cargo check right? so it's gonna try and compile electrsd for WASM. So you need to #[cfg(not(target_arch = "wasm32"))] and not depend on the thing in the Cargo.toml if the arch is WASM.

Tbh I'm not sure why secp256k1 depends on rand maybe we can disable a feature that prevents including rand? That would be much simpler.

I figured out the issue. Using [target.'cfg(not(target_arch = "wasm32"))'.dev-dependencies] in the crates/esplora/Cargo.toml didn't work until I enabled resolver = "2" in the top level workspace Cargo.toml. I would have expected it to already be using the new resolver since all our crates are set to edition = "2021", but apparently the resolver also needs to be set for the whole workspace.

https://blog.rust-lang.org/2021/03/25/Rust-1.51.0.html#cargos-new-feature-resolver

notmandatory · 2023-09-14T20:03:31Z

Rebased on master.

LLFourn

Thanks @notmandatory. Just to be clear I fully understand why it fails. I missed that u32::MAX was how we were disabling it.

ACK 7be6f86

LLFourn · 2023-09-19T01:16:04Z

.github/workflows/cont_integration.yml

@@ -38,7 +38,9 @@ jobs:
          cargo update -p flate2:1.0.27 --precise "1.0.26"
          cargo update -p reqwest --precise "0.11.18"
          cargo update -p h2 --precise "0.3.20"
-          cargo update -p rustls-webpki --precise "0.100.1"
+          cargo update -p rustls-webpki:0.100.3 --precise "0.100.1"
+          cargo update -p rustls-webpki:0.101.5 --precise "0.101.1"


Why do we depend on two slightly different versions of webpki?

Looks like rustls-webpki 0.100.x is required by the rustls version used by electrum-client.

And rustls-webpki 0.101.x is required directly by the version of minreq used by our dev-dependencies bitcoind and electrsd.

I think the ultimate solution for this will be to get all our dependencies to use the same major+minor version of rustls.

This fixes overflow error when calling update_tx_graph() from update_tx_graph_without_keychain().

…rsion 2 The resolver version must be set at the workspace level. See: https://blog.rust-lang.org/2021/03/25/Rust-1.51.0.html#cargos-new-feature-resolver

notmandatory · 2023-09-26T02:13:18Z

Rebased and fixed two more ci MSRV issues. Had to pin tokio-util and update the pinned version of rustls-webpki:0.101.6.

notmandatory · 2023-09-26T17:56:52Z

FYI, after I merged this PR the audit job failed due to the new dev dependency on the bitcoind, I'm not sure why it didn't fail when I created the PR. The issue should be fixed with a new bitcoind patch release after rust-bitcoin/bitcoind#141 is merged.

notmandatory added the bug Something isn't working label Sep 4, 2023

notmandatory added this to the 1.0.0-alpha.2 milestone Sep 4, 2023

notmandatory self-assigned this Sep 4, 2023

notmandatory mentioned this pull request Sep 4, 2023

Add cli esplora example #1040

Merged

4 tasks

notmandatory force-pushed the test/esplora_tests branch 2 times, most recently from b156772 to c161a21 Compare September 4, 2023 06:52

LLFourn reviewed Sep 5, 2023

View reviewed changes

notmandatory force-pushed the test/esplora_tests branch 2 times, most recently from 4d3e367 to 26f90a6 Compare September 6, 2023 01:59

notmandatory mentioned this pull request Sep 6, 2023

test: consolidate esplora blocking and async test code #1113

Open

nondiremanuel mentioned this pull request Sep 10, 2023

W37 Lib Team Call bitcoindevkit/.github#76

Closed

41 tasks

notmandatory force-pushed the test/esplora_tests branch 2 times, most recently from 47d09e6 to a42fb3d Compare September 14, 2023 20:02

notmandatory force-pushed the test/esplora_tests branch from a42fb3d to 7be6f86 Compare September 14, 2023 20:46

LLFourn approved these changes Sep 19, 2023

View reviewed changes

nondiremanuel mentioned this pull request Sep 23, 2023

W39 Lib Team Call bitcoindevkit/.github#78

Closed

40 tasks

notmandatory added 5 commits September 25, 2023 21:09

test(esplora): add async_ext and blocking_ext integration tests

4d1a9fd

fix(esplora): use saturating_add in update_tx_graph()

6a5c9d7

This fixes overflow error when calling update_tx_graph() from update_tx_graph_without_keychain().

ci: fix msrv dependency versions for rustls-webpki and zip

31d52e1

ci(esplora): fix wasm cargo check by setting workspace resolver to ve…

d35668e

…rsion 2 The resolver version must be set at the workspace level. See: https://blog.rust-lang.org/2021/03/25/Rust-1.51.0.html#cargos-new-feature-resolver

ci: fix MSRV build by pinning tokio-util to 0.7.8

bf9a425

notmandatory force-pushed the test/esplora_tests branch from 7be6f86 to bf9a425 Compare September 26, 2023 02:12

notmandatory merged commit 4a1b96d into bitcoindevkit:master Sep 26, 2023
12 checks passed

nondiremanuel mentioned this pull request Oct 6, 2023

W41 Lib Team Call bitcoindevkit/.github#80

Closed

39 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(esplora): use saturating_add in update_tx_graph() #1110

fix(esplora): use saturating_add in update_tx_graph() #1110

notmandatory commented Sep 4, 2023 •

edited

LLFourn commented Sep 4, 2023

notmandatory commented Sep 4, 2023

notmandatory commented Sep 4, 2023 •

edited

LLFourn Sep 5, 2023

notmandatory Sep 6, 2023

LLFourn Sep 5, 2023

notmandatory Sep 6, 2023

LLFourn Sep 6, 2023

notmandatory Sep 6, 2023

LLFourn Sep 6, 2023

notmandatory Sep 14, 2023 •

edited

notmandatory commented Sep 14, 2023

LLFourn left a comment •

edited

LLFourn Sep 19, 2023

notmandatory Sep 26, 2023

notmandatory commented Sep 26, 2023

notmandatory commented Sep 26, 2023

fix(esplora): use saturating_add in update_tx_graph() #1110

fix(esplora): use saturating_add in update_tx_graph() #1110

Conversation

notmandatory commented Sep 4, 2023 • edited

Description

Notes to the reviewers

Changelog notice

Checklists

All Submissions:

Bugfixes:

LLFourn commented Sep 4, 2023

notmandatory commented Sep 4, 2023

notmandatory commented Sep 4, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

notmandatory Sep 14, 2023 • edited

Choose a reason for hiding this comment

notmandatory commented Sep 14, 2023

LLFourn left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

notmandatory commented Sep 26, 2023

notmandatory commented Sep 26, 2023

notmandatory commented Sep 4, 2023 •

edited

notmandatory commented Sep 4, 2023 •

edited

notmandatory Sep 14, 2023 •

edited

LLFourn left a comment •

edited