Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: n-historical states feature flags #6538

Merged
merged 3 commits into from
Mar 20, 2024
Merged

feat: n-historical states feature flags #6538

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b3abebf
feat: n-historical states feature flags
twoeths Mar 12, 2024
d80f310
fix: change vitest config for the using keyword
twoeths Mar 13, 2024
590edef
chore: address PR comments
twoeths Mar 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
33 changes: 29 additions & 4 deletions packages/beacon-node/src/chain/chain.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ import {IExecutionEngine, IExecutionBuilder} from "../execution/index.js";
import {Clock, ClockEvent, IClock} from "../util/clock.js";
import {ensureDir, writeIfNotExist} from "../util/file.js";
import {isOptimisticBlock} from "../util/forkChoice.js";
import {BufferPool} from "../util/bufferPool.js";
import {BlockProcessor, ImportBlockOpts} from "./blocks/index.js";
import {ChainEventEmitter, ChainEvent} from "./emitter.js";
import {IBeaconChain, ProposerPreparationData, BlockHash, StateGetOpts, CommonBlockBody} from "./interface.js";
Expand Down Expand Up @@ -80,7 +81,11 @@ import {BlockRewards, computeBlockRewards} from "./rewards/blockRewards.js";
import {ShufflingCache} from "./shufflingCache.js";
import {StateContextCache} from "./stateCache/stateContextCache.js";
import {SeenGossipBlockInput} from "./seenCache/index.js";
import {CheckpointStateCache} from "./stateCache/stateContextCheckpointsCache.js";
import {InMemoryCheckpointStateCache} from "./stateCache/stateContextCheckpointsCache.js";
import {FIFOBlockStateCache} from "./stateCache/fifoBlockStateCache.js";
import {PersistentCheckpointStateCache} from "./stateCache/persistentCheckpointsCache.js";
import {DbCPStateDatastore} from "./stateCache/datastore/db.js";
import {FileCPStateDatastore} from "./stateCache/datastore/file.js";
import {SyncCommitteeRewards, computeSyncCommitteeRewards} from "./rewards/syncCommitteeRewards.js";

/**
Expand Down Expand Up @@ -239,9 +244,28 @@ export class BeaconChain implements IBeaconChain {
this.pubkey2index = cachedState.epochCtx.pubkey2index;
this.index2pubkey = cachedState.epochCtx.index2pubkey;

const stateCache = new StateContextCache({metrics});
const checkpointStateCache = new CheckpointStateCache({metrics});

const fileDataStore = opts.nHistoricalStatesFileDataStore ?? false;
const stateCache = this.opts.nHistoricalStates
? new FIFOBlockStateCache(this.opts, {metrics})
: new StateContextCache({metrics});
const checkpointStateCache = this.opts.nHistoricalStates
? new PersistentCheckpointStateCache(
{
metrics,
logger,
clock,
shufflingCache: this.shufflingCache,
getHeadState: this.getHeadState.bind(this),
bufferPool: new BufferPool(anchorState.type.tree_serializedSize(anchorState.node), metrics),
datastore: fileDataStore
? // debug option if we want to investigate any issues with the DB
new FileCPStateDatastore()
: // production option
new DbCPStateDatastore(this.db),
},
this.opts
)
: new InMemoryCheckpointStateCache({metrics});
const {checkpoint} = computeAnchorCheckpoint(config, anchorState);
stateCache.add(cachedState);
stateCache.setHeadState(cachedState);
Expand Down Expand Up @@ -335,6 +359,7 @@ export class BeaconChain implements IBeaconChain {

/** Populate in-memory caches with persisted data. Call at least once on startup */
async loadFromDisk(): Promise<void> {
await this.regen.init();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dumb question: Why don't we need to regen.init() in the BeaconChain constructor?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's a conventional place to do async init, see the this.opPool.fromPersisted(this.db) call below

await this.opPool.fromPersisted(this.db);
}

Expand Down
13 changes: 10 additions & 3 deletions packages/beacon-node/src/chain/forkChoice/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import {
ForkChoiceStore,
ExecutionStatus,
JustifiedBalancesGetter,
ForkChoiceOpts,
ForkChoiceOpts as RealForkChoiceOpts,
} from "@lodestar/fork-choice";
import {
CachedBeaconStateAllForks,
Expand All @@ -21,7 +21,10 @@ import {ChainEventEmitter} from "../emitter.js";
import {ChainEvent} from "../emitter.js";
import {GENESIS_SLOT} from "../../constants/index.js";

export type {ForkChoiceOpts};
export type ForkChoiceOpts = RealForkChoiceOpts & {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like the use of word "real" is a little unconventional here though I can't come up with a better name. Maybe RawForkChoiceOpts?

// for testing only
forkchoiceConstructor?: typeof ForkChoice;
};

/**
* Fork Choice extended with a ChainEventEmitter
Expand All @@ -47,7 +50,11 @@ export function initializeForkChoice(

const justifiedBalances = getEffectiveBalanceIncrementsZeroInactive(state);

return new ForkChoice(
// forkchoiceConstructor is only used for some test cases
// production code use ForkChoice constructor directly
const forkchoiceConstructor = opts.forkchoiceConstructor ?? ForkChoice;

return new forkchoiceConstructor(
config,

new ForkChoiceStore(
Expand Down
11 changes: 11 additions & 0 deletions packages/beacon-node/src/chain/options.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,17 @@ import {ArchiverOpts} from "./archiver/index.js";
import {ForkChoiceOpts} from "./forkChoice/index.js";
import {LightClientServerOpts} from "./lightClient/index.js";
import {ShufflingCacheOpts} from "./shufflingCache.js";
import {DEFAULT_MAX_BLOCK_STATES, FIFOBlockStateCacheOpts} from "./stateCache/fifoBlockStateCache.js";
import {PersistentCheckpointStateCacheOpts} from "./stateCache/persistentCheckpointsCache.js";
import {DEFAULT_MAX_CP_STATE_EPOCHS_IN_MEMORY} from "./stateCache/persistentCheckpointsCache.js";

export type IChainOptions = BlockProcessOpts &
PoolOpts &
SeenCacheOpts &
ForkChoiceOpts &
ArchiverOpts &
FIFOBlockStateCacheOpts &
PersistentCheckpointStateCacheOpts &
ShufflingCacheOpts &
LightClientServerOpts & {
blsVerifyAllMainThread?: boolean;
Expand All @@ -31,6 +36,8 @@ export type IChainOptions = BlockProcessOpts &
broadcastValidationStrictness?: string;
minSameMessageSignatureSetsToBatch: number;
archiveBlobEpochs?: number;
nHistoricalStates?: boolean;
nHistoricalStatesFileDataStore?: boolean;
};

export type BlockProcessOpts = {
Expand Down Expand Up @@ -103,4 +110,8 @@ export const defaultChainOptions: IChainOptions = {
// batching too much may block the I/O thread so if useWorker=false, suggest this value to be 32
// since this batch attestation work is designed to work with useWorker=true, make this the lowest value
minSameMessageSignatureSetsToBatch: 2,
nHistoricalStates: false,
nHistoricalStatesFileDataStore: false,
maxBlockStates: DEFAULT_MAX_BLOCK_STATES,
maxCPStateEpochsInMemory: DEFAULT_MAX_CP_STATE_EPOCHS_IN_MEMORY,
};
5 changes: 2 additions & 3 deletions packages/beacon-node/src/chain/stateCache/stateContextCheckpointsCache.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import {routes} from "@lodestar/api";
import {Metrics} from "../../metrics/index.js";
import {StateCloneOpts} from "../regen/interface.js";
import {MapTracker} from "./mapMetrics.js";
import {CheckpointStateCache as CheckpointStateCacheInterface, CacheItemType} from "./types.js";
import {CheckpointStateCache, CacheItemType} from "./types.js";

export type CheckpointHex = {epoch: Epoch; rootHex: RootHex};
const MAX_EPOCHS = 10;
Expand All @@ -16,9 +16,8 @@ const MAX_EPOCHS = 10;
* belonging to checkpoint
*
* Similar API to Repository
* TODO: rename to MemoryCheckpointStateCache in the next PR of n-historical states
*/
export class CheckpointStateCache implements CheckpointStateCacheInterface {
export class InMemoryCheckpointStateCache implements CheckpointStateCache {
private readonly cache: MapTracker<string, CachedBeaconStateAllForks>;
/** Epoch -> Set<blockRoot> */
private readonly epochIndex = new MapDef<Epoch, Set<string>>(() => new Set<string>());
Expand Down
4 changes: 2 additions & 2 deletions packages/beacon-node/src/chain/validation/block.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,13 +111,13 @@ export async function validateGossipBlock(
});
}

// getBlockSlotState also checks for whether the current finalized checkpoint is an ancestor of the block.
// use getPreState to reload state if needed. It also checks for whether the current finalized checkpoint is an ancestor of the block.
// As a result, we throw an IGNORE (whereas the spec says we should REJECT for this scenario).
// this is something we should change this in the future to make the code airtight to the spec.
// [IGNORE] The block's parent (defined by block.parent_root) has been seen (via both gossip and non-gossip sources) (a client MAY queue blocks for processing once the parent block is retrieved).
// [REJECT] The block's parent (defined by block.parent_root) passes validation.
const blockState = await chain.regen
.getBlockSlotState(parentRoot, blockSlot, {dontTransferCache: true}, RegenCaller.validateGossipBlock)
.getPreState(block, {dontTransferCache: true}, RegenCaller.validateGossipBlock)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the same to getBlockSlotState() except for it allows to reload state if needed
if block epoch is same to parent epoch, there is no need to dial to block slot
if block epoch > parent epoch, it'll use checkpoint state of block epoch. In regen.ts it'll use getBlockSlotState() in the end

lodestar/packages/beacon-node/src/chain/regen/regen.ts

Line 91 in 822446f

return this.getBlockSlotState(toHexString(cp.root), checkpointStartSlot, opts, regenCaller, allowDiskReload);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this will reduce number of calls to processSlotsToNearestCheckpoint() which will reduce epochTransitionByCaller metric

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's just the same, if parentEpoch < blockEpoch it'll get checkpoint state. The PrepareNextSlot should do the epoch transition for us anyway so epoch transition doesn't come from here

.catch(() => {
throw new BlockGossipError(GossipAction.IGNORE, {code: BlockErrorCode.PARENT_UNKNOWN, parentRoot});
});
Expand Down