Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Header MMR (One MMR To Rule Them All) #1716

Merged
merged 20 commits into from Oct 15, 2018

Conversation

antiochp
Copy link
Member

@antiochp antiochp commented Oct 10, 2018

  • Add support for a db backed MMR (hash file, data maintained in db).
  • Add additional MMRs to txhashset
    • header MMR (headers based on header_head)
    • sync MMR (headers based on sync_head)
  • txhashset::extending now extends header MMR (in addition to output, rangeproof, kernel)
  • txhashset::readonly_extending extends header MMR (in addition to output, rangeproof, kernel)
  • txhashset::header_extending extends header MMR in isolation (for header validation, where we do not yet have full blocks)
  • txhashset::sync_extending extends sync MMR only (for handling batches of headers during sync)
  • after receiving output, rangeproof, kernel MMRs from peer via txhashset.zip receiving node now re-builds header MMR based on headers in db prior to validating the reconstructed txhashset
  • on entering sync state we rebuild the sync MMR based on current header_head - this allows us to safely rewind and reapply headers as necessary based on current most work chain

None of this is consensus breaking, we just add the MMRs in this PR.
Consensus breaking changes will come in a separate PR.


TODO (this PR) -

  • Cleanup code.
  • Cleanup comment/doc warnings.
  • More testing.
  • Optimize rebuild_header_mmr() and rebuild_sync_mmr() or are they acceptable for now?
  • skip rewind() on "next header" for header_extending and sync_extending.
  • Header MMR tracks header_head - confirm this is ok if/when head and header_head differ.
    • "Catch up" on same chain if nodes falls behind.
    • Short forks where two blocks get mined simultaneously.
  • Support migrating an existing node
    • Create/backfill header MMRs as necessary on chain::init()

TODO (potentially separate PR) -

  • This adds a header MMR but we do not commit to the MMR root yet in block headers -
    • Replace header.previous with header.prev_root.
    • DB indices to allow us to find previous header easily?
    • Validate prev_root prior when validating blocks and headers.
    • What would be involved in hard-forking this in later?

Optimizations/Improvements to think about -

  • Skip rewind() on "next header" during header extension (we can live without this initially).
  • "Rebuild" sync MMR from header MMR by simply copying the files themselves (no need to actually rebuild the MMR from the hashes)
  • Can we extend the header MMR during header validation then simply reuse the updated header MMR when applying full blocks? (Right now we rewind and reapply a couple of times, first the validate header, then to validate and apply the block).

works with fast sync
not yet in place for initial header sync
rebuild header MMR after we validate full txhashset after download
@antiochp antiochp changed the title [WIP] The Header MMR (one MMR to rule them all) The Header MMR (One MMR To Rule Them All) Oct 15, 2018
@antiochp
Copy link
Member Author

Merging.

@antiochp antiochp merged commit 8b7a20f into mimblewimble:master Oct 15, 2018
antiochp added a commit to antiochp/grin that referenced this pull request Oct 15, 2018
* header MMR in use within txhashset itself
works with fast sync
not yet in place for initial header sync

* add the (currently unused) sync_head mmr

* use sync MMR during fast sync
rebuild header MMR after we validate full txhashset after download

* support missing header MMR (rebuild as necessary) for legacy nodes

* rename to HashOnly

* cleanup backend.append()

* simplify vec_backend to match simpler append api
antiochp added a commit that referenced this pull request Oct 15, 2018
* header MMR in use within txhashset itself
works with fast sync
not yet in place for initial header sync

* add the (currently unused) sync_head mmr

* use sync MMR during fast sync
rebuild header MMR after we validate full txhashset after download

* support missing header MMR (rebuild as necessary) for legacy nodes

* rename to HashOnly

* cleanup backend.append()

* simplify vec_backend to match simpler append api
@antiochp antiochp deleted the simple_header_mmr branch August 29, 2019 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants