feat(ShardAccessor): feat: enable concurrent ShardAccessor usage with mmap #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… mmap - Modify the ShardAccessor struct to use the existing mmapr field for concurrent access - Update the Read method to use mmapr when available, falling back to the data reader if not - Refactor the Blockstore method to reuse the mmapr field if it is already set, preventing unnecessary file opens - This change enables concurrent access to the ShardAccessor without opening the file multiple times, improving performance and resource usage.
renaynay
approved these changes
Apr 3, 2023
distractedm1nd
commented
Apr 3, 2023
Wondertan
requested changes
Apr 3, 2023
distractedm1nd
force-pushed
the
mmap-reader
branch
from
0b4d5ee
to
5690211
Compare
renaynay
reviewed
Apr 4, 2023
Wondertan
reviewed
Apr 4, 2023
Co-authored-by: Hlib Kanunnikov <hlibwondertan@gmail.com>
renaynay
approved these changes
Apr 4, 2023
distractedm1nd
added a commit
to celestiaorg/celestia-node
that referenced
this pull request
Apr 4, 2023
Closes #1514 . This PR enables accessor cache usage for GetCAR and GetDAH. This will allow shrexeds and shrexnd servers to only require opening the underlying file once (until removed from cache). If many peers request the EDS at once, the server previously needed to open the file each time. Will still require a dependency bump of our dagstore fork for the new test to not fail - so draft until then. Related: celestiaorg/dagstore#2 --------- Co-authored-by: rene <41963722+renaynay@users.noreply.github.com> Co-authored-by: Hlib Kanunnikov <hlibwondertan@gmail.com>
walldiss
pushed a commit
that referenced
this pull request
Jul 24, 2023
… mmap (#2) * feat(ShardAccessor): feat: enable concurrent ShardAccessor usage with mmap - Modify the ShardAccessor struct to use the existing mmapr field for concurrent access - add `io.Reader` factory, `Reader() io.Reader` produces readers that use mmapr when available, falling back to the data reader if not - Refactor the Blockstore method to reuse the mmapr field if it is already set, preventing unnecessary file opens - This change enables concurrent access to the ShardAccessor without opening the file multiple times, improving performance and resource usage. * remove comment * refactor: `io.Reader` factory, instead of implementing iface directly * Update accessor.go Co-authored-by: Hlib Kanunnikov <hlibwondertan@gmail.com> * wrapping error --------- Co-authored-by: Hlib Kanunnikov <hlibwondertan@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related: celestiaorg/celestia-node#1514. Will open a PR there that adds a test to verify this functionality in the LRU cache, and enables cache usage for GetCAR and GetDAH. This will allow shrexeds and shrexnd servers to only require opening the file once (until removed from cache)
This change enables concurrent access to the ShardAccessor without opening the file multiple times, improving performance and resource usage.