caches reed-solomon encoder/decoder instance #27510
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
steviez
previously approved these changes
Sep 1, 2022
carllin
reviewed
Sep 2, 2022
| @@ -346,6 +369,38 @@ impl Shredder { | |||
| } | |||
| } | |||
|
|
|||
| impl ReedSolomonCache { | |||
| const CAPACITY: usize = 4 * DATA_SHREDS_PER_FEC_BLOCK; | |||
There was a problem hiding this comment.
nit: A comment on why this should be a function of DATA_SHREDS_PER_FEC_BLOCK would be helpful
|
Waiting on upstream reed-solomon-erasure crate merge rust-rse/reed-solomon-erasure#104 first. |
behzadnouri
added a commit
to behzadnouri/solana
that referenced
this pull request
Sep 23, 2022
Need to pick up: rust-rse/reed-solomon-erasure#104 in order to unblock: solana-labs#27510
behzadnouri
added a commit
to behzadnouri/solana
that referenced
this pull request
Sep 23, 2022
Need to pick up: rust-rse/reed-solomon-erasure#104 in order to unblock: solana-labs#27510
behzadnouri
force-pushed
the
reed-solomon-cache
branch
from
57ec9e2
to
d4ff770
Compare
mergify
bot
dismissed stale reviews from sakridge and steviez
Pull request has been modified.
behzadnouri
added a commit
that referenced
this pull request
Sep 24, 2022
Need to pick up: rust-rse/reed-solomon-erasure#104 in order to unblock: #27510
mergify bot
pushed a commit
that referenced
this pull request
Sep 24, 2022
Need to pick up: rust-rse/reed-solomon-erasure#104 in order to unblock: #27510 (cherry picked from commit f02fe9c) # Conflicts: # ledger/Cargo.toml
behzadnouri
force-pushed
the
reed-solomon-cache
branch
from
d4ff770
to
8a837d0
Compare
ReedSolomon::new(...) initializes a matrix and an inversion-tree: https://github.com/rust-rse/reed-solomon-erasure/blob/eb1f66f47/src/core.rs#L450-L458 In order to cache this computation, this commit caches the reed-solomon encoder/decoder instance for each (data_shards, parity_shards) pair.
behzadnouri
force-pushed
the
reed-solomon-cache
branch
from
8a837d0
to
c90a970
Compare
mergify bot
added a commit
that referenced
this pull request
Sep 24, 2022
…#28048) * updates reed-solomon-erasure crate version to 6.0.0 (#28033) Need to pick up: rust-rse/reed-solomon-erasure#104 in order to unblock: #27510 (cherry picked from commit f02fe9c) # Conflicts: # ledger/Cargo.toml * removes mergify merge conflicts Co-authored-by: behzad nouri <behzadnouri@gmail.com>
|
With rust-rse/reed-solomon-erasure#104 merged in, this also shows improvement on the recovery end. Also slightly more shreds are recovered (maybe because of faster recovery code): |
mergify bot
pushed a commit
that referenced
this pull request
Sep 25, 2022
ReedSolomon::new(...) initializes a matrix and a data-decode-matrix cache: https://github.com/rust-rse/reed-solomon-erasure/blob/273ebbced/src/core.rs#L460-L466 In order to cache this computation, this commit caches the reed-solomon encoder/decoder instance for each (data_shards, parity_shards) pair. (cherry picked from commit f49beb0)
mergify bot
added a commit
that referenced
this pull request
Sep 26, 2022
caches reed-solomon encoder/decoder instance (#27510) ReedSolomon::new(...) initializes a matrix and a data-decode-matrix cache: https://github.com/rust-rse/reed-solomon-erasure/blob/273ebbced/src/core.rs#L460-L466 In order to cache this computation, this commit caches the reed-solomon encoder/decoder instance for each (data_shards, parity_shards) pair. (cherry picked from commit f49beb0) Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
ReedSolomon::new(...)initializes a matrix and a data-decode-matrix cache:https://github.com/rust-rse/reed-solomon-erasure/blob/273ebbced/src/core.rs#L460-L466
Current master code is redoing this computation for each batch.
Summary of Changes
In order to cache this computation, this commit caches the reed-solomon
encoder/decoder instance for each
(data_shards, parity_shards)pair.