Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement headers first synchronization #428

Merged
merged 251 commits into from
Nov 28, 2022

Conversation

markin-io
Copy link
Contributor

@markin-io markin-io commented Jun 9, 2022

Issue being fixed or feature implemented

Feature implements headers first synchronization in Dash wallet and refines a way transactions obtaining their metadata
during the synchronization process

What was done?

Headers sync

  • Implemented headers first synchronization
  • Implemented storage management for headers and their metadata
    • Storage holds last 5000 headers
    • Storage holds all headers metadata such as time and height to be used by transactions on later stage
  • Reworked a way of obtaining new chain height
  • Implemented retry logic for continuous headers sync (with DAPIStream.js)
  • Provided event about the headers sync progress (total/confirmed)

TX sync

  • Got rid of DDOSing getTransaction and getBlockHeader requests to DAPI
  • Implemented obtaining of the TX metadata from block headers via merkle blocks
  • Provide TX time in metadata
  • Implemented TX sync progress event
  • Refactoring and cleanup

dash-spv

  • Optimizations and bugfixes
  • Introducing hash->height and height->hash indices
  • Reworking to support pruned chains
  • Improving asynchronous chain assembly process

Additional changes

  • Added supplemental options for Storage for ease of debugging
  • Used wasm-x11-hash instead of the js implementation
  • Minor test execution and linting improvements
  • Reworked plugins execution sequence (onStart all plugins first and only then execute all of them instead of onStart -> execute per each)
  • Minor JSON validation improvements for Storage
  • Logging and error handling improvements
  • Implemented DAPIStream wrapper for indefinitely running streams with retry logic

Leftovers

TX Sync

  • Integration tests simulating multiple streams

SPV

// Will be done in a separate ticket

  • Reorgs
  • Verification against merkle blocks
  • Enable consensus checks
  • Chain validation function

Storage

  • Pruning of headers metadata once transactions acknowledged it
  • Workflow with skipSynchronizationBeforeHeight
  • platform-test-suite with storage enabled
  • reorg safe storage saving procedure

Misc

  • Network aliasing uncerainties
  • Update dashcore to v0.19.40
  • Fix TypeScript definitions
  • Use wasm-x11 in dapi-client headers tests

How Has This Been Tested?

  • Unit and integration tests
  • Manually against testnet
  • CI

Breaking Changes

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have made corresponding changes to the documentation

For repository code-owners and collaborators only

  • I have assigned this pull request to a milestone

@lgtm-com
Copy link

lgtm-com bot commented Jun 9, 2022

This pull request introduces 1 alert when merging 7175fde into 83d4e96 - view on LGTM.com

new alerts:

  • 1 for Unused variable, import, function or class

@lgtm-com
Copy link

lgtm-com bot commented Jun 13, 2022

This pull request introduces 2 alerts when merging da64713 into 7a5715c - view on LGTM.com

new alerts:

  • 1 for Useless assignment to local variable
  • 1 for Unused variable, import, function or class

@lgtm-com
Copy link

lgtm-com bot commented Jun 14, 2022

This pull request introduces 3 alerts when merging 0d58ee7 into 7a5715c - view on LGTM.com

new alerts:

  • 2 for Unused variable, import, function or class
  • 1 for Useless assignment to local variable

Copy link
Member

@shumkov shumkov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please take a look at the comments

@markin-io markin-io changed the title feat(wallet-lib): integrate with block headers stream feat(wallet-lib): implement headers first synchronization Nov 28, 2022
Copy link
Member

@shumkov shumkov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's huge! Good job!

@markin-io markin-io merged commit 0f50af3 into master Nov 28, 2022
@markin-io markin-io deleted the feat/integrate-with-block-headers-stream branch November 28, 2022 16:12
@shumkov shumkov changed the title feat(wallet-lib): implement headers first synchronization feat: implement headers first synchronization Dec 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants