Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: use multiple threads when exporting snapshots #3331

Merged
merged 7 commits into from
Aug 7, 2023

Conversation

lemmih
Copy link
Contributor

@lemmih lemmih commented Aug 7, 2023

Summary of changes

Changes introduced in this pull request:

  • Exporting a snapshot involves two CPU-bound tasks: Walking the graph, and encoding the ForestCAR.zst file. This PR moves each task to a separate thread, nearly doubling the throughput.

Benchmark (three runs each):

Command Mean [s] Min [s] Max [s] Relative
export calibnet (par-buffer) 27.646 ± 0.295 27.324 27.904 1.00
export calibnet (baseline) 48.516 ± 0.458 47.999 48.873 1.75

Reference issue to close (if applicable)

Closes

Other information and links

Change checklist

  • I have performed a self-review of my own code,
  • I have made corresponding changes to the documentation,
  • I have added tests that prove my fix is effective or that my feature works (if possible),
  • I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

@lemmih lemmih marked this pull request as ready for review August 7, 2023 10:57
@lemmih lemmih requested a review from a team as a code owner August 7, 2023 10:57
@lemmih lemmih requested review from LesnyRumcajs, sudo-shashank and aatifsyed and removed request for a team August 7, 2023 10:57
Copy link
Contributor

@aatifsyed aatifsyed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

src/chain/mod.rs Show resolved Hide resolved
@lemmih lemmih enabled auto-merge August 7, 2023 17:32
@lemmih lemmih added this pull request to the merge queue Aug 7, 2023
Merged via the queue into main with commit 6b1013a Aug 7, 2023
20 checks passed
@lemmih lemmih deleted the lemmih/stream-par-buffer branch August 7, 2023 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants