Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize Syncer #89

Merged
merged 14 commits into from
Aug 6, 2020
Merged

Optimize Syncer #89

merged 14 commits into from
Aug 6, 2020

Conversation

patrick-ogrady
Copy link
Contributor

@patrick-ogrady patrick-ogrady commented Aug 6, 2020

Related Issue: coinbase/mesh-cli#87

This PR greatly improves sync performance by changing syncing to be continuous instead of discrete (syncing in 1000 block chunks). Most performance gains come from keeping our IO resources in high use (previously, we wouldn't attempt to store anything while fetching the block range). I will post a chart describing the magnitude of sync speedup when merging this into the rosetta-cli.

As a side effect, we greatly reduce memory usage by avoiding caching up to 1000 blocks in memory before processing (when the default concurrency is used, this is a memory usage reduction of 99%).

Changes

  • Make syncer continuous instead of discrete
  • Improve syncer test coverage
  • Change BlockConcurrency to SyncConcurrency
  • Remove BlockConcurrency from Fetcher
  • Cleanup logic in syncer and in tests

@coveralls
Copy link

coveralls commented Aug 6, 2020

Pull Request Test Coverage Report for Build 3279

  • 87 of 99 (87.88%) changed or added relevant lines in 2 files are covered.
  • 1 unchanged line in 1 file lost coverage.
  • Overall coverage increased (+4.3%) to 69.833%

Changes Missing Coverage Covered Lines Changed/Added Lines %
syncer/configuration.go 4 8 50.0%
syncer/syncer.go 83 91 91.21%
Files with Coverage Reduction New Missed Lines %
syncer/syncer.go 1 86.49%
Totals Coverage Status
Change from base Build 3150: 4.3%
Covered Lines: 2169
Relevant Lines: 3106

💛 - Coveralls

@patrick-ogrady patrick-ogrady changed the title [WIP] Optimize Syncer Optimize Syncer Aug 6, 2020
@patrick-ogrady patrick-ogrady requested review from niallo and a team August 6, 2020 17:00
syncer/syncer.go Outdated Show resolved Hide resolved
syncer/syncer.go Show resolved Hide resolved
syncer/syncer.go Show resolved Hide resolved
Copy link
Contributor

@qiwu7 qiwu7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

qiwu7
qiwu7 previously approved these changes Aug 6, 2020
@patrick-ogrady patrick-ogrady merged commit a0b262f into master Aug 6, 2020
@patrick-ogrady patrick-ogrady deleted the patrick/syncer-optimization branch August 6, 2020 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants