Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[reconciler] Performance Overhaul #258

Merged
merged 11 commits into from
Dec 1, 2020
Merged

Conversation

patrick-ogrady
Copy link
Contributor

@patrick-ogrady patrick-ogrady commented Dec 1, 2020

Related: https://github.com/coinbase/rosetta-sdk-go/tree/patrick/priority-lock

This PR introduces performance optimizations in the reconciler that reduce lock contention in the syncing loop. This PR also migrates BalanceStorage, BlockStorage, and CoinStorage to use github.com/neilotoole/errgroup (in practice, significantly reduces memory usage without impacting performance).

Changes

reconciler

  • introduce utils.ShardedMap to track currently used accounts
  • update tests
    • fix failing test where context.Canceled returned instead of reconciliation fails
  • make enqueuing block changes async

statefulsyncer

  • change statefulsyncer default pruning frequency to 10 minutes

storage

  • explicitly set errgroup limit with runtime.NumCPU
  • ensure BlockStorage is not using a global database transaction
  • migrate BlockStorage to use github.com/neilotoole/errgroup
  • migrate CoinStorage to use github.com/neilotoole/errgroup

@coveralls
Copy link

coveralls commented Dec 1, 2020

Pull Request Test Coverage Report for Build 12826

  • 76 of 88 (86.36%) changed or added relevant lines in 5 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.05%) to 78.183%

Changes Missing Coverage Covered Lines Changed/Added Lines %
reconciler/reconciler.go 63 75 84.0%
Totals Coverage Status
Change from base Build 12702: 0.05%
Covered Lines: 7891
Relevant Lines: 10093

💛 - Coveralls

@sidhujag
Copy link

sidhujag commented Dec 1, 2020

we are able to sync our node now with official SDK, I will retest again with these changes but so far so good, memory being held in check as well. (We have increased ours due to the database change size issue but can play around there once these changes are in)

Copy link

@yfl92 yfl92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants