Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable faster sync #3108

Merged
merged 10 commits into from Nov 26, 2019
Merged

Enable faster sync #3108

merged 10 commits into from Nov 26, 2019

Conversation

@antiochp
Copy link
Member

antiochp commented Nov 4, 2019

Non-consensus related implementation for RFC mimblewimble/grin-rfcs#29

  • refactor vec_backend so we can use it outside of tests
  • introduce a "hash only" vec_backend for the accumulator
  • introduce bitmap_accumulator -
    • MMR backed by a hash only vec backend
    • leaves are 1024 bit chunks of the leafset bitmap
    • it is relatively small, keep it in memory only
    • no need to store data in leaves (hash only) as data is maintained in the existing leafset
  • implement "rewind" logic so we handle bitmap_accumulator correctly during a fork/reorg
    • it is not append-only as the bitmap updates over time
  • build new bitmap accumulator root for each new block
    • log only (no consensus changes)

We will eventually be able to use this bitmap accumulator root to verify fragments of the leafset against subtrees of the existing output PMMR. This will allow us to verify "fast sync" data in chunks, without needing to download the full txhashset.zip which is increasingly becoming a bottleneck and single point of failure during initial fast sync.


TODO -

  • test coverage for various bitmap accumulator scenarios (particularly rewind)
  • test coverage for bitmap fragment serialization (endianness etc)
@antiochp antiochp added this to the 3.0.0 milestone Nov 4, 2019
@antiochp antiochp changed the title add bitmap accumulator Enable faster sync Nov 12, 2019
@antiochp antiochp force-pushed the antiochp:bitmap_acc branch from b131422 to 84cd632 Nov 13, 2019
@antiochp antiochp marked this pull request as ready for review Nov 13, 2019
@antiochp antiochp self-assigned this Nov 13, 2019
@antiochp antiochp force-pushed the antiochp:bitmap_acc branch 2 times, most recently from 8f9ebe6 to 6fe57ad Nov 14, 2019
@antiochp

This comment has been minimized.

Copy link
Member Author

antiochp commented Nov 15, 2019

We now log this for every new block -

20191115 11:48:41.272 DEBUG grin_chain::types - validate roots: 00000c3194d8 at 435707, 
output_root: 4856008b7c31, output pmmr: 4856008b7c31 (bitmap: 7a488eb915b8, merged: 454f61a2a6df)

It would be good to get this merged and have multiple nodes deployed with this code so we can verify the merged root looks acceptable across a wider sample of nodes on the network.

Note: We are not currently using the merged root for validation, simply logging it for visibility.
The RFC proposal is to move to validating this root at the HF.

@antiochp antiochp requested review from hashmap, tromp and yeastplume Nov 15, 2019
@quentinlesceller

This comment has been minimized.

Copy link
Member

quentinlesceller commented Nov 15, 2019

@antiochp taking a look today this weekend.

@tromp

This comment has been minimized.

Copy link
Contributor

tromp commented Nov 15, 2019

chain/src/txhashset/txhashset.rs Outdated Show resolved Hide resolved
chain/src/txhashset/txhashset.rs Outdated Show resolved Hide resolved
core/src/ser.rs Outdated Show resolved Hide resolved
Copy link
Contributor

tromp left a comment

i added some comments and questions to the code diffs.

@antiochp antiochp force-pushed the antiochp:bitmap_acc branch 2 times, most recently from a3bae98 to 57d5782 Nov 18, 2019
antiochp added 4 commits Oct 10, 2019
refactor vec backend so we can use it outside of tests
introduce a "hash only" vec backend for the accumulator
antiochp added 6 commits Nov 12, 2019
…ation
@antiochp antiochp force-pushed the antiochp:bitmap_acc branch from b1b6da1 to 7fd4e37 Nov 25, 2019
@antiochp

This comment has been minimized.

Copy link
Member Author

antiochp commented Nov 26, 2019

I'm going to merge this now. It adds the accumulator without relying on it yet. And adds logging around the accumulator which I'd like to start seeing on multiple nodes on mainnet and floonet sooner rather than later.
There are no consensus changes in this PR - these are coming soon as part of the scheduled HF2.

@antiochp antiochp merged commit 11ac7d8 into mimblewimble:master Nov 26, 2019
10 checks passed
10 checks passed
mimblewimble.grin #20191125.2 succeeded
Details
mimblewimble.grin (linux api/util/store) linux api/util/store succeeded
Details
mimblewimble.grin (linux chain/core/keychain) linux chain/core/keychain succeeded
Details
mimblewimble.grin (linux pool/p2p/src) linux pool/p2p/src succeeded
Details
mimblewimble.grin (linux release) linux release succeeded
Details
mimblewimble.grin (linux servers) linux servers succeeded
Details
mimblewimble.grin (macos release) macos release succeeded
Details
mimblewimble.grin (macos test) macos test succeeded
Details
mimblewimble.grin (windows release) windows release succeeded
Details
mimblewimble.grin (windows test) windows test succeeded
Details
@antiochp antiochp mentioned this pull request Dec 11, 2019
@antiochp antiochp deleted the antiochp:bitmap_acc branch Jan 9, 2020
garyyu added a commit to garyyu/grin that referenced this pull request Jan 14, 2020
* add bitmap accumulator
refactor vec backend so we can use it outside of tests
introduce a "hash only" vec backend for the accumulator

* get core tests passing

* initial test coverage for bitmap_accumulator

* better test coverage for bitmap accumulator and cleanup code

* refactor txhashset roots, call validate() on roots during block validation

* fix store tests

* log the "merged" root when validating roots

* cleanup, revise based on feedback

* cleanup

* rework it to pass explicit size into bitmap accumulator when applying
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.