feat: use local checkpoint state for holesky-rescue by twoeths · Pull Request #7509 · ChainSafe/lodestar

twoeths · 2025-02-28T03:12:53Z

Motivation

we already have local checkpoint states in either db or file, this PR use one of them to save syncing time the next time the node is restarted
see Holesky Retro Features: Introduce mechanisms to force Lodestar to be on a minority chain #7504 (comment)

Description

2 new init unfinalized state options:

new --lastPersistedCheckpointState, true by default to load from the last safe persisted checkpoint state. And the last safe persisted checkpoint state is considered unfinalized
use --checkpointState option: this is old option and support finalized state only, this PR supports booting from unfinalized state similar to --lastPersistedCheckpointState ==> to quickly sync a new node, we can use this option. Then remove it and the next time node will use --lastPersistedCheckpointState option by default
we can configure one of our nodes with chain.nHistoricalStatesFileDataStore = true
then feed other nodes with a persisted "safe checkpoint state" from there in ~/checkpoint_states folder
a persisted checkpoint state is consider to be safe to boot if
- it should be the checkpoint state that's unique in its epoch
- its last processed block slot should be at epoch boundary or last slot of previous epoch
- state slot should be at epoch boundary
- state slot should be equal to epoch * SLOTS_PER_EPOCH

other options to boot from stateArchived or checkpointSyncUrl are considered finalized states including wssCheckpoint. It's not possible to use wssCheckpoint option with unfinalized state for now.

TODO

this is for holesky-rescue, consider supporting this for unstable branch too
update document in that case

twoeths · 2025-02-28T03:41:58Z

was able to start from local checkpoint state in db

Feb 28 03:27:54 feat1-lg1k-hzax41 beacon_run.sh[4045147]: Feb-28 03:27:54.845[]              verbose: Loading local checkpoint state from db checkpoint=0x3f998475da108d0f83cb40becef580c6e35280a2983deda527bfbf8ebf0df807:116689
Feb 28 03:27:55 feat1-lg1k-hzax41 beacon_run.sh[4045147]: Feb-28 03:27:55.117[]                 info: Loaded local checkpoint state from db checkpoint=0x3f998475da108d0f83cb40becef580c6e35280a2983deda527bfbf8ebf0df807:116689
Feb 28 03:28:17 feat1-lg1k-hzax41 beacon_run.sh[4045147]: Feb-28 03:28:17.942[]                 info: Initializing beacon from a valid checkpoint state slot=3734048, epoch=116689, stateRoot=0x909e6612fdfef407d2e737e8c89eaa13491685493ff5a820ee0db865b3c4ff21, isWithinWeakSubjectivityPeriod=true

twoeths · 2025-02-28T06:46:22Z

was able to load local checkpoint state from file

Feb 28 06:43:30 feat1-md64-hzax41 beacon_run.sh[777935]: Feb-28 06:43:30.295[]                 info: Loading local checkpoint state from file path=/home/devops/beacon/checkpoint_states/0x24c6010000000000db693d0a9b6b76b4b7b7478cfbd6d8c1c39557adc13b13192e5de30d4e4a8363
Feb 28 06:43:30 feat1-md64-hzax41 beacon_run.sh[777935]: Feb-28 06:43:30.976[]                 info: Loaded local checkpoint state from file path=/home/devops/beacon/checkpoint_states/0x24c6010000000000db693d0a9b6b76b4b7b7478cfbd6d8c1c39557adc13b13192e5de30d4e4a8363
Feb 28 06:43:34 feat1-md64-hzax41 beacon_run.sh[777935]: Feb-28 06:43:34.369[]                 info: Initializing beacon from a valid checkpoint state slot=3720320, epoch=116260, stateRoot=0xa5ce19fd4bdced2f1d8040dad942b0a360fbe07f9db718b7a886e2ccb3a8afdb, isWithinWeakSubjectivityPeriod=true

This reverts commit 4d2bd7b.

twoeths · 2025-03-06T09:37:57Z

I deployed the latest commit to feat1 (docker only, 2 nodes uses chain.nHistoricalStatesFileDataStore)
I restarted the whole group 3 times (x5 nodes = 15 times) and it all works fine

twoeths · 2025-03-07T02:27:29Z

-    startEpoch: Math.min(computeEpochAtSlot(local.headSlot), remote.finalizedEpoch),
+    // startEpoch: Math.min(computeEpochAtSlot(local.headSlot), remote.finalizedEpoch),
+    // for holesky-rescue, we don't want to sync from finalizedEpoch which is too far away (115967)
+    startEpoch: computeEpochAtSlot(local.headSlot),


this is not correct in case we sync from the last finalized checkpoint
once we got on a wrong fork, we cannot recover

for example this is the wrong fork

8527:Mar-06 17:56:30.653[chain] verbose: Block processed slot=3711858, root=0xb6b21a75a1587c3af9037946a564a58548edeac52d028b0abdd7ec9b0baeed04, delaySec=839094.6530001163 8558:Mar-06 17:56:31.076[chain] verbose: Block processed slot=3711860, root=0x7b0aeb3fd90bd78e2f70c38f64fc7d94abeda215581793c3d111fdac325ab26a, delaySec=839071.0759999752 9586:Mar-06 17:56:59.940[chain] verbose: Block processed slot=3711964, root=0xd0165ccec1ebba1a08c1c69a070acf9df638f7acca1922591f530683c541c945, delaySec=837851.9400000572 9634:Mar-06 17:57:02.911[chain] verbose: Block processed slot=3711974, root=0xfe1d75c5b393401b2dd01fe85fb103a1485055f47c7299089ec7da6aecf2fff1, delaySec=837734.9110000134

correct one should be

4291:Mar-06 17:56:00.593[chain] verbose: Block processed slot=3711858, root=0xb6b21a75a1587c3af9037946a564a58548edeac52d028b0abdd7ec9b0baeed04, delaySec=839064.5929999352 4326:Mar-06 17:56:01.020[chain] verbose: Block processed slot=3711860, root=0x7b0aeb3fd90bd78e2f70c38f64fc7d94abeda215581793c3d111fdac325ab26a, delaySec=839041.0199999809 4395:Mar-06 17:56:06.181[chain] verbose: Block processed slot=3711901, root=0xe372ed53b4badda9eb1f8db5549649a77886c88e91aa92f5ff3b0c934562bc1c, delaySec=838554.1809999943 4463:Mar-06 17:56:09.686[chain] verbose: Block processed slot=3711913, root=0xcd2f0403e62139d1e77440d66df9abacc2c706e45527e95cc51569c9d328de9a, delaySec=838413.6860001087 4509:Mar-06 17:56:10.090[chain] verbose: Block processed slot=3711917, root=0x69a749989af18cf9f77f433b4af321ee3d0f2ffaecde9c84c86cddd7fc13f2b8, delaySec=838366.0899999142

so need to distinguish 2 cases:

if sync from finalized checkpoint state: startEpoch: Math.min(computeEpochAtSlot(local.headSlot), remote.finalizedEpoch) (same logic)

if sync from unfinalized checkpoint state: startEpoch: Math.min(computeEpochAtSlot(local.headSlot), Math.max(remote.finalizedEpoch, computeEpochAtSlot(trustedHead.slot)))

nflaig

LGTM

**Motivation** - implement an api to get a node synced asap **Description** - new api: `eth/v1/lodestar/persisted_checkpoint_state` to return a state based on an optional `rootHex:epoch` param - if not specified, return the latest safe checkpoint state - a node need to specify `--checkpointState` from the previous PR #7509 **Test** - [x] `curl -H "Accept: application/octet-stream" http://localhost:9596/eth/v1/lodestar/persisted_checkpoint_state -o latest_checkpoint_state.ssz` - [x] `curl -H "Accept: application/octet-stream" http://localhost:9596/eth/v1/lodestar/persisted_checkpoint_state?checkpoint_id=0x4f4d4c1b81141fe77a4a1c6a376dbe64ed9baa8f123664195e4f710c9fc4238d:118936 -o state_epoch_118936.ssz` --------- Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com> Co-authored-by: Nico Flaig <nflaig@protonmail.com>

**Motivation** - we already have local checkpoint states in either db or file, this PR use one of them to save syncing time the next time the node is restarted - see #7504 (comment) **Description** 2 new init unfinalized state options: - new `--lastPersistedCheckpointState`, true by default to load from the last safe persisted checkpoint state. And the last safe persisted checkpoint state is considered unfinalized - use `--checkpointState` option: this is old option and support finalized state only, this PR supports booting from unfinalized state similar to `--lastPersistedCheckpointState` ==> to quickly sync a new node, we can use this option. Then remove it and the next time node will use `--lastPersistedCheckpointState` option by default - we can configure one of our nodes with `chain.nHistoricalStatesFileDataStore = true` - then feed other nodes with a persisted "safe checkpoint state" from there in `~/checkpoint_states` folder - a persisted checkpoint state is consider to be safe to boot if - it should be the checkpoint state that's unique in its epoch - its last processed block slot should be at epoch boundary or last slot of previous epoch - state slot should be at epoch boundary - state slot should be equal to epoch * SLOTS_PER_EPOCH other options to boot from `stateArchived` or `checkpointSyncUrl` are considered finalized states including `wssCheckpoint`. It's not possible to use `wssCheckpoint` option with unfinalized state for now. **TODO** - this is for `holesky-rescue`, consider supporting this for `unstable` branch too - update document in that case --------- Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com>

**Motivation** - implement an api to get a node synced asap **Description** - new api: `eth/v1/lodestar/persisted_checkpoint_state` to return a state based on an optional `rootHex:epoch` param - if not specified, return the latest safe checkpoint state - a node need to specify `--checkpointState` from the previous PR #7509 **Test** - [x] `curl -H "Accept: application/octet-stream" http://localhost:9596/eth/v1/lodestar/persisted_checkpoint_state -o latest_checkpoint_state.ssz` - [x] `curl -H "Accept: application/octet-stream" http://localhost:9596/eth/v1/lodestar/persisted_checkpoint_state?checkpoint_id=0x4f4d4c1b81141fe77a4a1c6a376dbe64ed9baa8f123664195e4f710c9fc4238d:118936 -o state_epoch_118936.ssz` --------- Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com> Co-authored-by: Nico Flaig <nflaig@protonmail.com>

See #7509 (comment)

@twoeths

**Motivation** This was a feature we developed for [rescuing Holesky](https://blog.chainsafe.io/lodestar-holesky-rescue-retrospective/) as part of #7501 to quickly sync nodes to head during a period of long non-finality (~3 weeks). While it's unlikely we will have such a long period of non-finality on mainnet, this feature is still useful to have for much shorter periods and testing purposes on devnets. It is now part of [Ethereum protocol hardening](https://github.com/eth-clients/diamond) mitigations described [here](https://github.com/eth-clients/diamond/blob/main/mitigations/nfin-checkpoint-001.md) > Ordinary checkpoint sync begins from the latest finalized checkpoint (block and/or state). As an escape hatch during non-finality, it is useful to have the ability to checkpoint sync from an unfinalized checkpoint. A client implementing this mitigation MUST support checkpoint sync from an arbitrary non-finalized checkpoint state. We will support this with the exception that our checkpoint state needs to be an epoch boundary checkpoint. **Description** The main feature of this PR is to allow initializing a node from an unfinalized checkpoint state either retrieved locally or from a remote source. This behavior is disabled by default but can be enabled by either adding - the `--lastPersistedCheckpointState` flag to load from the last safe persisted checkpoint state stored locally - or `--unsafeCheckpointState` to provide a file path or url to an unfinalized checkpoint state to start syncing from which can be used with new endpoint `GET /eth/v1/lodestar/persisted_checkpoint_state` to sync from a remote node or by sharing states from `checkpoint_states` folder Both of these options are not safe to use on a network that recently finalized an epoch and must only be considered if syncing from last finalized checkpoint state is unfeasible. An unfinalized checkpoint state persisted locally is only considered to be safe to boot if - it's the only checkpoint in it's epoch to avoid ambiguity from forks - its last processed block slot is at an epoch boundary or last slot of previous epoch - state slot is at an epoch boundary - state slot is equal to `epoch * SLOTS_PER_EPOCH` But even if these criteria are met, there is chance that the node will end up on a minority chain as it will not be able to pivot to another chain that conflicts with the checkpoint state it was initialized from. Other existing flags (like `--checkpointState`) are unchanged by this PR and will continue to expect a finalized checkpoint state. Previous PRs #7509, #7541, #7542 not merged to unstable are included. Closes #7963 cc @twoeths --------- Co-authored-by: twoeths <10568965+twoeths@users.noreply.github.com> Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com>

twoeths force-pushed the te/start_from_local_ws_state branch from 36b577e to e189e52 Compare February 28, 2025 06:35

twoeths force-pushed the te/start_from_local_ws_state branch from 6e939fb to 46543d5 Compare February 28, 2025 09:17

twoeths added 21 commits March 4, 2025 15:40

feat: use local checkpoint state

7c5f1cb

fix: init forkchoice from non-finalized anchor state

7ef9521

fix: do not persist non-finalized local cp state to stateArchive db

7dd69e4

fix: initializeForkChoice - only change finalizedCheckpoint

555af56

feat: implement getFinalizedBlockSlot() for forkchoice

b5b73c0

chore: more logs initBeaconState()

c9a7b0f

feat: sync from local head

4d17449

fix: startEpoch of Head range sync

c0b1395

fix: ignore FORKCHOICE_ERROR_UNKNOWN_ANCESTOR check

c95bab8

Revert "fix: ignore FORKCHOICE_ERROR_UNKNOWN_ANCESTOR check"

31e2186

This reverts commit 4d2bd7b.

feat: initialize forkchoice from unfinalized state

3761e82

fix: add finalized proto block to ProtoArray

dff2b64

fix: startEpoch of getRangeSyncTarget

46c4b3e

fix: catch getDependentRoot

861955f

fix: processChainSegment to process blocks after head

df23e05

feat: lastPersistedCheckpointState flag

0d7b088

fix: getLastModifiedFile to return file name

dd33190

fix: getLastModifiedFile()

e45e6ac

fix: prioritize lastPersistedCheckpointState over checkpointState

84327b7

fix: pass dataDir to initBeaconState()

1141634

fix: only load from safe checkpoint state

e898a9b

twoeths force-pushed the te/start_from_local_ws_state branch from 1afbddc to e898a9b Compare March 4, 2025 09:05

twoeths added 4 commits March 4, 2025 16:14

fix: revert range sync

7fd1296

chore: refactor readLatest() to readLatestSafe()

e734895

feat: turn on lastPersistedCheckpointState by default

6402b44

fix: only boot from Current Root Checkpoint State

2062e68

twoeths added 3 commits March 5, 2025 11:15

fix: retry with checkpointState after lastPersistedCheckpointState

c665209

fix: getLatestSafeDatastoreKey

065ccb5

fix: also boot from Previous Root Checkpoint State (PRCS)

4528396

twoeths marked this pull request as ready for review March 5, 2025 11:37

twoeths requested a review from a team as a code owner March 5, 2025 11:37

nflaig reviewed Mar 5, 2025

View reviewed changes

Comment thread packages/cli/src/cmds/beacon/options.ts Outdated

Comment thread packages/cli/src/cmds/beacon/options.ts Outdated

Comment thread packages/cli/src/cmds/beacon/initBeaconState.ts Outdated

twoeths added 3 commits March 6, 2025 09:10

fix: state is finalizing if using checkpointSyncUrl

95344c0

chore: change log level when closing beacon node

ca867d5

fix: isFinalized, more logs and address PR comments

4b78e34

twoeths changed the title ~~feat: use local checkpoint state~~ feat: use local checkpoint state for holesky-rescue Mar 6, 2025

twoeths commented Mar 7, 2025

View reviewed changes

fix: head sync - do not always start from local head

34a12f7

nflaig approved these changes Mar 7, 2025

View reviewed changes

nflaig merged commit edab7f0 into holesky-rescue Mar 7, 2025

nflaig deleted the te/start_from_local_ws_state branch March 7, 2025 18:38

philknows mentioned this pull request Mar 7, 2025

Lodestar Holesky Rescue Overview #7503

Closed

twoeths mentioned this pull request Mar 10, 2025

feat: holesky-resuce - get persisted checkpoint state api #7541

Merged

2 tasks

nflaig mentioned this pull request Jun 13, 2025

Bring sync from unfinalized to unstable #7963

Closed

This was referenced Oct 17, 2025

feat: sync from unfinalized checkpoint state #8527

Merged

chore: simplify start epoch in head chain range sync #8535

Merged

nflaig added a commit that referenced this pull request Oct 17, 2025

Do not always start from local head in head sync

4087b35

See #7509 (comment)

Navigation Menu

Uh oh!

Uh oh!

feat: use local checkpoint state for holesky-rescue - #7509

feat: use local checkpoint state for holesky-rescue#7509
nflaig merged 32 commits into
holesky-rescuefrom
te/start_from_local_ws_state

twoeths commented Feb 28, 2025 •

edited

Loading

Uh oh!

twoeths commented Feb 28, 2025

Uh oh!

twoeths commented Feb 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

twoeths commented Mar 6, 2025

Uh oh!

twoeths Mar 7, 2025

Uh oh!

nflaig left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

twoeths commented Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

twoeths commented Feb 28, 2025

Uh oh!

twoeths commented Feb 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

twoeths commented Mar 6, 2025

Uh oh!

twoeths Mar 7, 2025

Choose a reason for hiding this comment

Uh oh!

nflaig left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

twoeths commented Feb 28, 2025 •

edited

Loading