Parallelize loading from S3 for better performance #4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vgrichina
added a commit
to fastnear/fast-near
that referenced
this pull request
May 13, 2022
It contains improvements on loading speed near/near-lake-framework-js#4
It allows to build code when installing npm package from source
frol
approved these changes
May 13, 2022
There was a problem hiding this comment.
@vgrichina Thanks for looking into it! Let's address the comments and merge it
src/s3fetchers.ts
Outdated
| @@ -21,12 +21,13 @@ import { normalizeBlockHeight, parseBody } from "./utils"; | |||
| export async function listBlocks( | |||
| client: S3Client, | |||
| bucketName: string, | |||
| startAfter: BlockHeight | |||
| startAfter: BlockHeight, | |||
| limit = 10 | |||
There was a problem hiding this comment.
Let's default to 200 since it will make less requests, and boosts the throughput quite a bit.
@khorolets Let's make this parameter configurable on Rust side as well and update the default to 200 (somehow we used 100 there, but choose to use 10 in JS version)
There was a problem hiding this comment.
looks like 200 might be a bit suboptimal when dealing with more meaty blocks, might need experiments on user's side to tune – e.g. at block #46661963 it seems that 100 is working better
maybe just needs another change to avoid blocking for all of them to be loaded
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note that this is relatively naive method which unnecessarily waits for one batch to complete before starting another batch load.
However it still seems to improve performance significantly.
Results in improvement of from about 2 blocks/second to 12+ blocks/second on my machine.
When measured at Hetzner box I use: