Use memory efficient toHex in pubkey2index map #3561
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dapplion
requested review from
g11tech,
mpetrunic,
twoeths and
wemeetagain
as code owners
Codecov Report
@@ Coverage Diff @@
## master #3561 +/- ##
=======================================
Coverage 37.47% 37.47%
=======================================
Files 311 311
Lines 8291 8291
Branches 1282 1282
=======================================
Hits 3107 3107
Misses 5036 5036
Partials 148 148 |
|
Code Climate has analyzed commit f5b920c and detected 0 issues on this pull request. View more on Code Climate. |
Performance Report✔️ no performance regression detected Full benchmark results
|
g11tech
reviewed
Jan 3, 2022
g11tech
approved these changes
Jan 3, 2022
wemeetagain
reviewed
Jan 3, 2022
| @@ -0,0 +1,215 @@ | |||
| export type TestRunnerMemoryOpts<T> = { | |||
There was a problem hiding this comment.
When you're ready, maybe this should be pulled into its own package. I see you're using it in ssz too here
There was a problem hiding this comment.
Agree it should eventually, tho it still needs more testing.
twoeths
approved these changes
Jan 4, 2022
dapplion
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
From the benchmarks in #3446 it's shown that concatenated strings take significant more memory than a string produced with Buffer.toString(). NodeJS's Buffer uses C++ bindings to produce the hex strings thus the different result.
For a network with 250_000 the concatenated strings for the pubkeys alone take 1400 * 250_000 = 350 MB.
Description
This PR should be followed by a general replacement of toHexString by this memory efficient approach. However, this PR does this minimal change first since pubkey2index takes the bulk of memory usage for concatenated strings.