-
Notifications
You must be signed in to change notification settings - Fork 702
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Merged by Bors] - Implement API for block rewards #2628
Conversation
TODO before this is ready to merge:
|
Update: I've added an SSE event to the |
I'm going to add the number of skipped slots from the parent to the reward information, because rewards at skipped slots are usually higher (and they should be). This will be a breaking change to the API between Lighthouse and blockprint but I'll update both together. |
3d6a5fe
to
189c31a
Compare
189c31a
to
7bf038a
Compare
I'd been getting memory leaks* running large queries against this API and I think I've finally found the cause. Profiling with dhat-rs shows huge amounts of memory being allocated by the tree hash cache, on the order of 370GB in 30 minutes (12GB per minute). This is exacerbated by running on a machine with 32 cores, in which case GNU malloc will use up to 64 memory arenas. As a workaround setting I'm planning to write a new generic |
d8f0bd0
to
b23269a
Compare
Rebased on #2863, although it hasn't entirely solved the memory issues on high-core machines so I'm going to continue using a |
## Issue Addressed Successor to #2431 ## Proposed Changes * Add a `BlockReplayer` struct to abstract over the intricacies of calling `per_slot_processing` and `per_block_processing` while avoiding unnecessary tree hashing. * Add a variant of the forwards state root iterator that does not require an `end_state`. * Use the `BlockReplayer` when reconstructing states in the database. Use the efficient forwards iterator for frozen states. * Refactor the iterators to remove `Arc<HotColdDB>` (this seems to be neater than making _everything_ an `Arc<HotColdDB>` as I did in #2431). Supplying the state roots allow us to avoid building a tree hash cache at all when reconstructing historic states, which saves around 1 second flat (regardless of `slots-per-restore-point`). This is a small percentage of worst-case state load times with 200K validators and SPRP=2048 (~15s vs ~16s) but a significant speed-up for more frequent restore points: state loads with SPRP=32 should be now consistently <500ms instead of 1.5s (a ~3x speedup). ## Additional Info Required by #2628
## Issue Addressed Successor to #2431 ## Proposed Changes * Add a `BlockReplayer` struct to abstract over the intricacies of calling `per_slot_processing` and `per_block_processing` while avoiding unnecessary tree hashing. * Add a variant of the forwards state root iterator that does not require an `end_state`. * Use the `BlockReplayer` when reconstructing states in the database. Use the efficient forwards iterator for frozen states. * Refactor the iterators to remove `Arc<HotColdDB>` (this seems to be neater than making _everything_ an `Arc<HotColdDB>` as I did in #2431). Supplying the state roots allow us to avoid building a tree hash cache at all when reconstructing historic states, which saves around 1 second flat (regardless of `slots-per-restore-point`). This is a small percentage of worst-case state load times with 200K validators and SPRP=2048 (~15s vs ~16s) but a significant speed-up for more frequent restore points: state loads with SPRP=32 should be now consistently <500ms instead of 1.5s (a ~3x speedup). ## Additional Info Required by #2628
b23269a
to
a549888
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
bors r+ |
## Proposed Changes Add an API endpoint for retrieving detailed information about block rewards. For information on usage see [the docs](https://github.com/sigp/lighthouse/blob/block-rewards-api/book/src/api-lighthouse.md#lighthouseblock_rewards), and the source.
bors r- gonna batch |
Canceled. |
bors r+ |
## Proposed Changes Add an API endpoint for retrieving detailed information about block rewards. For information on usage see [the docs](https://github.com/sigp/lighthouse/blob/block-rewards-api/book/src/api-lighthouse.md#lighthouseblock_rewards), and the source.
Pull request successfully merged into unstable. Build succeeded: |
Proposed Changes
Add an API endpoint for retrieving detailed information about block rewards.
For information on usage see the docs, and the source.