Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Space-efficient archive node #3626

Merged

Conversation

michaelsproul
Copy link
Member

Proposed Changes

Builds on tree-states to drastically reduce the size of archive nodes with SPRP=32. This is achieved using two main optimisations:

  • Avoid storing the duplicated immutable parts of the validator in the PartialBeaconStates of the DB. The immutable parts of the validator are its public key and withdrawal credentials.
  • Apply zstd compression to all PartialBeaconStates stored in the DB: this is effective at compressing the remaining fields that are likely to be similar, e.g. the mutable parts of state.validators, the state.balances, and the eth1 data votes.

Additional Info

My plan is to clean this up and merge it into #3206 along with #3583, which will give us tree-states + all the new database optimisations in one schema migration. To ease backwards-compatibility I'm going to investigate how to make the new schemas for restore points and frozen blocks optional, so that old nodes can keep running as they are, but new nodes and early adopters can switch to the more efficient schema.

@michaelsproul michaelsproul added work-in-progress PR is a work-in-progress optimization Something to make Lighthouse run more efficiently. labels Oct 4, 2022
@michaelsproul michaelsproul merged commit ff26c80 into sigp:tree-states Oct 19, 2022
@michaelsproul
Copy link
Member Author

I've merged this down into tree-states but will leave the branch up for anyone relying on it (just @alex88 I think).

@alex88
Copy link
Contributor

alex88 commented Oct 19, 2022

@michaelsproul I can just switch to that branch to continue testing it, not a problem! Thank you for thinking about it

@michaelsproul
Copy link
Member Author

@alex88 I've just made a bunch of backwards-incompat changes on tree-states, so for your existing node you should keep using this branch. The main tree-states branch is also going to undergo a bunch of other renovations in the next few days, so isn't really a stable target for new nodes either

@alex88
Copy link
Contributor

alex88 commented Oct 19, 2022

I've cloned it into my fork, just in case you change your mind :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
optimization Something to make Lighthouse run more efficiently. work-in-progress PR is a work-in-progress
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants