Use binary diff to store checkpoint state

[twoeths]

Problem description

Right now the finalized state slot 3710944 takes 257.61 MB. The holesky-branch has to only keep the last 100 epochs for the user which is not great

Solution description

instead of storing the whole state, we only store validators diff and state (without validators) diff using @chainsafe/xdelta3-node

import { describe, expect, it } from "vitest";
import fs from "fs";
import { formatBytes } from "@lodestar/utils";
import {decodeSync, encodeSync} from "@chainsafe/xdelta3-node";
import {holeskyChainConfig} from "@lodestar/config/networks";
import { createChainForkConfig } from "@lodestar/config";
import { getForkFromStateBytes, getStateTypeFromBytes } from "../src/util/sszBytes";
import { ssz } from "@lodestar/types";


describe("State diff", () => {
  const dir = ".";
  const statePath = `${dir}/holesky_state_3768287.ssz`;
  const finalizedStatePath = `${dir}/holesky_finalized_state_Feb_25.ssz`;
  const config = createChainForkConfig(holeskyChainConfig);
  it("load state and check validators size", () => {
    const stateBytes = fs.readFileSync(statePath);
    console.log("@@@ state bytes", formatBytes(stateBytes.length));
    const fork = getForkFromStateBytes(config, stateBytes);
    const forkName = config.getForkName(fork);
    console.log("@@@ fork", forkName);
    const stateType = getStateTypeFromBytes(config, stateBytes);
    const state = stateType.deserializeToViewDU(stateBytes);
    console.log("@@@ state", state.slot);
    const validatorsBytes = state.validators.serialize();
    console.log("@@@ validators bytes", formatBytes(validatorsBytes.length));

    const finalizedStateBytes = fs.readFileSync(finalizedStatePath);
    console.log("@@@ finalized state bytes", formatBytes(finalizedStateBytes.length));
    const finalizedStateType = getStateTypeFromBytes(config, finalizedStateBytes);
    const finalizedState = finalizedStateType.deserializeToViewDU(finalizedStateBytes);
    console.log("@@@ finalized state", finalizedState.slot);
    const finalizedValidatorsBytes = finalizedState.validators.serialize();
    console.log("@@@ finalized validators bytes", formatBytes(finalizedValidatorsBytes.length));

    const fullStateDelta = encodeSync(finalizedStateBytes, stateBytes);
    console.log("@@@ state delta", formatBytes(fullStateDelta.length));

    let start = Date.now();
    const validatorDelta = encodeSync(finalizedValidatorsBytes, validatorsBytes);
    console.log("@@@ validator delta", formatBytes(validatorDelta.length), Date.now() - start, "ms");
    start = Date.now();
    const validatorsBytes2 = decodeSync(finalizedValidatorsBytes, validatorDelta);
    console.log("@@@ apply delta in", Date.now() - start, "ms");
    expect(Buffer.compare(validatorsBytes, validatorsBytes2)).toBe(0);

    state.validators = ssz.phase0.Validators.defaultViewDU();
    state.commit();
    finalizedState.validators = ssz.phase0.Validators.defaultViewDU();
    finalizedState.commit();
    const stateWoValidatorsBytes = state.serialize();
    const finalizedStateWoValidatorsBytes = finalizedState.serialize();
    console.log("@@@ state without validators bytes", formatBytes(stateWoValidatorsBytes.length));
    console.log("@@@ finalized state without validators bytes", formatBytes(finalizedStateWoValidatorsBytes.length));
    const stateDelta = encodeSync(finalizedStateWoValidatorsBytes, stateWoValidatorsBytes);
    console.log("@@@ state delta", formatBytes(stateDelta.length));
  })
});

on our typical infrastructure it prints out

@@@ state bytes 257.65 MB
@@@ fork bellatrix
@@@ state 3768288
@@@ validators bytes 221.86 MB
@@@ finalized state bytes 257.61 MB
@@@ finalized state 3710944
@@@ finalized validators bytes 221.85 MB
@@@ state delta 21.69 MB
@@@ validator delta 8.17 MB 2749 ms
@@@ apply delta in 289 ms
@@@ state without validators bytes 35.79 MB
@@@ finalized state without validators bytes 35.77 MB
@@@ state delta 13.36 MB

so validator delta is just < 8.2 MB while state delta is < 13.36 MB, in total we can store < 22MB instead of 257MB
the down side is that it takes almost 3s to compute delta, need to consider worker thread or any native solution to avoid that

Additional context

No response

[twoeths]

looks like separating validators + remaining data does not help much

• full state diff = 21.69 MB
• validators diff + other data diff = 8.17 + 13.36 = 21.53 MB

[twoeths]

looks like separating validators + remaining data does not help much

the reason is there are 1611970 / 1922616 validators modified