[r2r] spv storage optimization #1 #1585

borngraced · 2022-12-21T14:48:08Z

This PR introduces new SPVConf structures with some fields that's needed to optimize stored block headers when spv proof is enabled

Main Optimization in this PR is allowing node to set max headers that needs to be stored(max_stored_block_headers) and where to start fetching block headers from(block_header).

pub struct SPVConf {
    /// Where to start block headers sync from.
    pub block_header: SPVBlockHeader,
    /// Max number of block headers to be stored in db, when reached, old header will be deleted.
    pub max_stored_block_headers: Option<NonZeroU64>,
    /// The parameters that specify how the coin block headers should be validated. If None,
    /// headers will be saved in DB without validation, can be none if the coin's RPC server is trusted.
    pub validation_params: Option<BlockHeaderValidationParams>,
}

Signed-off-by: borngraced <samiodev@icloud.com>

# Conflicts: # mm2src/coins/utxo/utxo_builder/utxo_arc_builder.rs # mm2src/coins/utxo/utxo_builder/utxo_coin_builder.rs

Signed-off-by: borngraced <samiodev@icloud.com>

shamardy

Thanks for the PR! Just a few comments to start the review, I will have more comments in next iterations.

mm2src/coins/utxo/utxo_builder/utxo_arc_builder.rs

mm2src/mm2_bitcoin/spv_validation/src/helpers_validation.rs

Signed-off-by: borngraced <samiodev@icloud.com>

shamardy

Great progress! Next review iteration.
@borngraced Can you please add a test similar to test_btc_block_header_sync but with optimization? you can use a recent retarget header as the starting block header and max_stored_block_headers=RETARGETING_INTERVAL. This test will not be ignored since the block headers sync time will be short.

mm2src/coins/utxo/utxo_block_header_storage/sql_block_header_storage.rs

mm2src/mm2_bitcoin/spv_validation/src/conf.rs

mm2src/mm2_bitcoin/spv_validation/src/helpers_validation.rs

mm2src/mm2_bitcoin/spv_validation/src/conf.rs

mm2src/mm2_bitcoin/spv_validation/src/work.rs

mm2src/mm2_test_helpers/src/for_tests.rs

mm2src/coins/utxo/utxo_tests.rs

mm2src/mm2_bitcoin/spv_validation/src/storage.rs

Signed-off-by: borngraced <samiodev@icloud.com>

shamardy

Next review iteration!

mm2src/mm2_main/tests/mm2_tests/mm2_tests_inner.rs

mm2src/mm2_bitcoin/spv_validation/src/conf.rs

mm2src/mm2_test_helpers/src/for_tests.rs

mm2src/coins/utxo/utxo_builder/utxo_arc_builder.rs

mm2src/mm2_main/tests/mm2_tests/mm2_tests_inner.rs

mm2src/coins/utxo/utxo_builder/utxo_arc_builder.rs

mm2src/mm2_bitcoin/spv_validation/src/conf.rs

mm2src/coins/utxo/utxo_tests.rs

Signed-off-by: borngraced <samiodev@icloud.com>

mm2src/mm2_test_helpers/src/for_tests.rs

mm2src/mm2_bitcoin/spv_validation/src/conf.rs

Signed-off-by: borngraced <samiodev@icloud.com>

shamardy

Next Iteration! I believe the next review will be the last one :)

mm2src/coins/utxo/utxo_tests.rs

mm2src/mm2_bitcoin/spv_validation/src/conf.rs

mm2src/coins/utxo/utxo_builder/utxo_arc_builder.rs

Signed-off-by: borngraced <samiodev@icloud.com>

shamardy

I believe this will be the last review iteration :)
Only some minor changes left

mm2src/coins/utxo/utxo_builder/utxo_arc_builder.rs

mm2src/mm2_bitcoin/spv_validation/src/conf.rs

mm2src/mm2_test_helpers/src/for_tests.rs

mm2src/mm2_bitcoin/spv_validation/src/lib.rs

mm2src/mm2_test_helpers/src/for_tests.rs

mm2src/mm2_bitcoin/spv_validation/src/helpers_validation.rs

mm2src/mm2_bitcoin/spv_validation/src/storage.rs

mm2src/mm2_bitcoin/spv_validation/src/work.rs

shamardy

Great work! Thanks for your patience with all the fixes @borngraced on this very tricky PR :)

shamardy · 2023-01-30T11:51:02Z

@sergeyboyko0791 Can you please take a look at this PR?

sergeyboyko0791

Thank you for the great enhancement!
I have multiple comments

mm2src/coins/utxo/utxo_builder/utxo_arc_builder.rs

mm2src/coins/utxo/utxo_tests.rs

mm2src/mm2_bitcoin/spv_validation/src/conf.rs

mm2src/mm2_main/tests/mm2_tests/mm2_tests_inner.rs

sergeyboyko0791 · 2023-01-31T12:56:06Z

mm2src/mm2_test_helpers/src/for_tests.rs

+        "spv_conf": {
+            "starting_block_header": {
+                "height": 764064,
+                "hash": "00000000000000000006da48b920343944908861fa05b28824922d9e60aaa94d",
+                "bits": 386375189,
+                "time": 1668986059,
+            },
+            "max_stored_block_headers": 3000,
+            "validation_params": {
+                "difficulty_check": true,
+                "constant_difficulty": false,
+                "difficulty_algorithm": "Bitcoin Mainnet"
+            }
        }


Doesn't this break the backward compatibility?
cc: @shamardy

The coins repo should be updated before merging this. Only BTC have spv configuration now in the coins file, and I think this change will make it easier to get the starting block header info from any block explorer unlike how it used to be with block header hex.

Thanks for the info! Hope there will not be any problem with it

Signed-off-by: borngraced <samiodev@icloud.com>

sergeyboyko0791

One more change please

mm2src/coins/utxo/utxo_builder/utxo_arc_builder.rs

Signed-off-by: borngraced <samiodev@icloud.com>

shamardy

Reapprove! @borngraced can you please post in the related issue the changes required to the coins configs to be merged with this?

shamardy · 2023-02-10T12:56:05Z

@ca333 I believe this can be merged to dev. Will require an update to BTC coin config according to this #1483 (comment).

borngraced added 14 commits December 14, 2022 14:09

save dev state — replaced enabled_spv_proof with SPVConf structure

a9e776e

Signed-off-by: borngraced <samiodev@icloud.com>

save dev state

47f76ad

Signed-off-by: borngraced <samiodev@icloud.com>

impl get_total_block_headers_count_from_storage

2731b25

Signed-off-by: borngraced <samiodev@icloud.com>

save dev state — impl remove_block_headers_from_storage

275b5f3

Signed-off-by: borngraced <samiodev@icloud.com>

save dev state

64fb8c3

Signed-off-by: borngraced <samiodev@icloud.com>

optimized spv block headers storage limit and starting block

2824d47

Signed-off-by: borngraced <samiodev@icloud.com>

removed SPVConf from UtxoCoinField

c7f1afa

Signed-off-by: borngraced <samiodev@icloud.com>

save dev state — block_headers_limit_to_remove

866a0b5

Signed-off-by: borngraced <samiodev@icloud.com>

save dev state — impl block_headers_limit_to_remove

f9ada57

Signed-off-by: borngraced <samiodev@icloud.com>

save dev state — test_get_total_block_headers_from_storage unit test

462e3a6

Signed-off-by: borngraced <samiodev@icloud.com>

save dev state — test_remove_block_headers_from_storage unit test

7c56268

Signed-off-by: borngraced <samiodev@icloud.com>

Merge remote-tracking branch 'origin/dev' into spv-optimizations

08864ef

# Conflicts: # mm2src/coins/utxo/utxo_builder/utxo_arc_builder.rs # mm2src/coins/utxo/utxo_builder/utxo_coin_builder.rs

optimize spv storage

862553e

Signed-off-by: borngraced <samiodev@icloud.com>

minor changes to get_last_block_height

3ce2b23

Signed-off-by: borngraced <samiodev@icloud.com>

borngraced requested a review from shamardy December 21, 2022 14:48

borngraced self-assigned this Dec 21, 2022

borngraced linked an issue Dec 21, 2022 that may be closed by this pull request

Implement SPV with minimal storage requirements and fast block headers sync time #1483

Open

fix validate header fn

4b8a654

Signed-off-by: borngraced <samiodev@icloud.com>

borngraced changed the title ~~[wip] spv storage optimization 1#~~ [r2r] spv storage optimization #1 Dec 22, 2022

fix tests

9584f86

Signed-off-by: borngraced <samiodev@icloud.com>

borngraced changed the title ~~[r2r] spv storage optimization #1~~ [wip] spv storage optimization #1 Dec 22, 2022

borngraced added 3 commits December 22, 2022 22:07

save dev state — headers difficulty validation

9c09e19

Signed-off-by: borngraced <samiodev@icloud.com>

save dev state — calculate_retarget_height and unit test

ff3cf63

Signed-off-by: borngraced <samiodev@icloud.com>

minor changes

4ccf4ac

Signed-off-by: borngraced <samiodev@icloud.com>

borngraced changed the title ~~[wip] spv storage optimization #1~~ [r2r] spv storage optimization #1 Dec 22, 2022

minor changes

828fad6

Signed-off-by: borngraced <samiodev@icloud.com>

shamardy requested changes Dec 26, 2022

View reviewed changes

borngraced changed the title ~~[r2r] spv storage optimization #1~~ [wip] spv storage optimization #1 Dec 26, 2022

borngraced added 2 commits December 26, 2022 18:34

fixed block header validation check

c218712

Signed-off-by: borngraced <samiodev@icloud.com>

fix doc comments

55bfce4

Signed-off-by: borngraced <samiodev@icloud.com>

borngraced changed the title ~~[wip] spv storage optimization #1~~ [r2r] spv storage optimization #1 Jan 23, 2023

borngraced added 2 commits January 23, 2023 15:29

fix review notes and refactoring

44224ff

Signed-off-by: borngraced <samiodev@icloud.com>

fix docs

d380e5c

Signed-off-by: borngraced <samiodev@icloud.com>

shamardy requested changes Jan 24, 2023

View reviewed changes

borngraced added 3 commits January 24, 2023 13:10

removed retarget header from mainnet_next_block_bits params

b024b2e

Signed-off-by: borngraced <samiodev@icloud.com>

test_spv_conf_with_verification unit test

649bfe1

Signed-off-by: borngraced <samiodev@icloud.com>

impl validate starting block header from rpc!

1e9b0e1

Signed-off-by: borngraced <samiodev@icloud.com>

shamardy requested changes Jan 25, 2023

View reviewed changes

fix review notes

f1e3e72

Signed-off-by: borngraced <samiodev@icloud.com>

shamardy reviewed Jan 26, 2023

View reviewed changes

mm2src/mm2_test_helpers/src/for_tests.rs Outdated Show resolved Hide resolved

mm2src/mm2_bitcoin/spv_validation/src/conf.rs Show resolved Hide resolved

borngraced added 2 commits January 26, 2023 16:47

use correct bit for genesis block - btc testnet

3e346eb

Signed-off-by: borngraced <samiodev@icloud.com>

refactor remove_headers_up_to_height_sql

f957ddc

Signed-off-by: borngraced <samiodev@icloud.com>

shamardy requested changes Jan 26, 2023

View reviewed changes

mm2src/coins/utxo/utxo_tests.rs Outdated Show resolved Hide resolved

mm2src/mm2_bitcoin/spv_validation/src/conf.rs Outdated Show resolved Hide resolved

mm2src/coins/utxo/utxo_builder/utxo_arc_builder.rs Outdated Show resolved Hide resolved

fix review notes.

e4d05df

Signed-off-by: borngraced <samiodev@icloud.com>

shamardy requested changes Jan 27, 2023

View reviewed changes

fix reveiw notes

7239b0e

shamardy previously approved these changes Jan 30, 2023

View reviewed changes

shamardy requested a review from sergeyboyko0791 January 30, 2023 11:51

sergeyboyko0791 suggested changes Jan 31, 2023

View reviewed changes

fix minor review notes

3b41a0a

Signed-off-by: borngraced <samiodev@icloud.com>

borngraced dismissed shamardy’s stale review via 3b41a0a January 31, 2023 21:40

sergeyboyko0791 suggested changes Feb 1, 2023

View reviewed changes

mm2src/coins/utxo/utxo_builder/utxo_arc_builder.rs Outdated Show resolved Hide resolved

use map_to_mm for error handling

82d1bcb

Signed-off-by: borngraced <samiodev@icloud.com>

shamardy approved these changes Feb 10, 2023

View reviewed changes

borngraced mentioned this pull request Feb 10, 2023

Implement SPV with minimal storage requirements and fast block headers sync time #1483

Open

ca333 approved these changes Feb 13, 2023

View reviewed changes

ca333 merged commit e60bf64 into dev Feb 13, 2023

borngraced mentioned this pull request Feb 17, 2023

[release] 1.0.0-beta #1576

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[r2r] spv storage optimization #1 #1585

[r2r] spv storage optimization #1 #1585

borngraced commented Dec 21, 2022 •

edited

shamardy left a comment

shamardy left a comment

shamardy left a comment

shamardy left a comment

shamardy left a comment

shamardy left a comment

shamardy commented Jan 30, 2023

sergeyboyko0791 left a comment

sergeyboyko0791 Jan 31, 2023

shamardy Jan 31, 2023

sergeyboyko0791 Jan 31, 2023 •

edited

sergeyboyko0791 left a comment

shamardy left a comment

shamardy commented Feb 10, 2023

[r2r] spv storage optimization #1 #1585

[r2r] spv storage optimization #1 #1585

Conversation

borngraced commented Dec 21, 2022 • edited

shamardy left a comment

Choose a reason for hiding this comment

shamardy left a comment

Choose a reason for hiding this comment

shamardy left a comment

Choose a reason for hiding this comment

shamardy left a comment

Choose a reason for hiding this comment

shamardy left a comment

Choose a reason for hiding this comment

shamardy left a comment

Choose a reason for hiding this comment

shamardy commented Jan 30, 2023

sergeyboyko0791 left a comment

Choose a reason for hiding this comment

sergeyboyko0791 Jan 31, 2023

Choose a reason for hiding this comment

shamardy Jan 31, 2023

Choose a reason for hiding this comment

sergeyboyko0791 Jan 31, 2023 • edited

Choose a reason for hiding this comment

sergeyboyko0791 left a comment

Choose a reason for hiding this comment

shamardy left a comment

Choose a reason for hiding this comment

shamardy commented Feb 10, 2023

borngraced commented Dec 21, 2022 •

edited

sergeyboyko0791 Jan 31, 2023 •

edited