feat: prune persisted cp states - #7510
Conversation
|
works fine on 
sometimes it's only 9 persisted checkpoint states because epoch states in memory = 4 also checked on disc: |
|
do we wanna re-open this and target unstable branch to merge it there as well later on? |
| * During nft state of Holesky in Feb 2025, lodestar stores ~250MB per epoch and it's not sustainable to keep all states on disk. | ||
| * It's not likely to have reorgs that go back to 10 epochs ago, so we only keep 10 epochs on disk. | ||
| */ | ||
| export const DEFAULT_MAX_CP_STATE_ON_DISK = 10; |
There was a problem hiding this comment.
I bumped this number up to 100 on the holesky-rescue branch just to make it a bit safer and 25GB is really not a lot considering we have an increase of ~56GB today. I notified people in the telegram group to use the new image if they run low on strorage.
But I talked with @wemeetagain about your changes and we concluded this is not really safe behavior so we might need to come up with a better strategy. One idea was to use ERA files to compress states that are older than last 10 (or 100) epochs, I don't have any numbers but that might reduce storage increase quite a bit.
I also like the idea to just use file storage and let people manually delete states but that sounds a bit tricky as well, like how do you figure out which states to delete..
Would be interesting to get your view @twoeths on this
There was a problem hiding this comment.
One idea was to use ERA files to compress states that are older than last 10 (or 100) epochs
yes we should explore it
I also like the idea to just use file storage and let people manually delete states but that sounds a bit tricky as well, like how do you figure out which states to delete..
the benefit of that is to share checkpoint state to other node, and we can view checkpoint states visually. The file name is checkpoint's serialized data so we know the epoch + root there, and it's sorted by persisted time. Also I don't think leveldb is efficient as a hot db, ie to push data and remove that later since it has to deal with multiple levels inside
There was a problem hiding this comment.
just think about that again, we can just store state diff work in #7005 to save disc space, it's the best strategy to me. The state.validators does not seem to change much over epochs, we can diff that separately to save the diff file. Need to see how much the state is without validators
There was a problem hiding this comment.
just think about that again, we can just store state diff work in #7005 to save disc space, it's the best strategy to me.
yeah wanted to ask you this as well if that was a possibility once we get it working
**Motivation** Last change from #7501 which we implemented because persisted checkpoint states are added each epoch during non-finality and never pruned until the chain finalizes again. It turns out this is not sustainable if we have multiple weeks of non-finality since it takes up hundreds of GB of disk space and many nodes don't have sufficient disk space to handle this. The long term solution is to store states more efficiently but for now we should at least have a option to enable pruning, there is also always the options to clean up the `checkpoint_states` folder manually. **Description** This PR adds a new flag `--chain.maxCPStateEpochsOnDisk` to enable pruning of persisted checkpoint states. By default we don't prune any persistent checkpoint states as it's not safe to delete them during long non-finality as we don't know the state of the chain and there could be a deep (hundreds of epochs) reorg if there two competing chains with similar weight but we wouldn't have a close enough state to pivot to this chain and instead require a resync from last finalized checkpoint state which could be very far in the past. Previous PR #7510 --------- Co-authored-by: twoeths <10568965+twoeths@users.noreply.github.com>
2 participants
Motivation
Description
delete persisted checkpoint state in add() function, by default only store max 10 persisted checkpoint state
see Holesky Retro Features: Introduce mechanisms to force Lodestar to be on a minority chain聽#7504 (comment)