Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: allow blob archival for bigger time periods #6393

Merged
merged 3 commits into from
Feb 16, 2024
Merged

feat: allow blob archival for bigger time periods #6393

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ece8109
feat: allow blob archival for bigger time periods
g11tech Feb 5, 2024
1b29088
fix spellcheck and spellings
g11tech Feb 6, 2024
1873034
change cli flag and desc
g11tech Feb 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
31 changes: 19 additions & 12 deletions packages/beacon-node/src/chain/archiver/archiveBlocks.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@ export async function archiveBlocks(
lightclientServer: LightClientServer,
logger: Logger,
finalizedCheckpoint: CheckpointHex,
currentEpoch: Epoch
currentEpoch: Epoch,
archiveBlobEpochs?: number
): Promise<void> {
// Use fork choice to determine the blocks to archive and delete
// getAllAncestorBlocks response includes the finalized block, so it's also moved to the cold db
Expand Down Expand Up @@ -81,19 +82,25 @@ export async function archiveBlocks(
}

// Delete expired blobs
// Keep only `[max(GENESIS_EPOCH, current_epoch - MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS), current_epoch]`
// Keep only `[current_epoch - max(MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS, archiveBlobEpochs)]
// if archiveBlobEpochs set to Infinity do not prune`
nflaig marked this conversation as resolved.
Show resolved Hide resolved
if (finalizedPostDeneb) {
const blobSidecarsMinEpoch = currentEpoch - config.MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS;
if (blobSidecarsMinEpoch >= config.DENEB_FORK_EPOCH) {
const slotsToDelete = await db.blobSidecarsArchive.keys({lt: computeStartSlotAtEpoch(blobSidecarsMinEpoch)});
if (slotsToDelete.length > 0) {
await db.blobSidecarsArchive.batchDelete(slotsToDelete);
logger.verbose(
`blobSidecars prune: batchDelete range ${slotsToDelete[0]}..${slotsToDelete[slotsToDelete.length - 1]}`
);
} else {
logger.verbose(`blobSidecars prune: no entries before epoch ${blobSidecarsMinEpoch}`);
if (archiveBlobEpochs !== Infinity) {
const blobsArchiveWindow = Math.max(config.MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS, archiveBlobEpochs ?? 0);
const blobSidecarsMinEpoch = currentEpoch - blobsArchiveWindow;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we are fine with this being a negative number, in that case, do we even need to handle Infinity explicitly?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thats a good point. Would just get skipped as normal, although the log output would be funny...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

although the log output would be funny

It is not that bad

blobSidecars prune: no entries before epoch -Infinity

but I guess the other log is more explicit about it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its good to be explicit not just for code correctness but also because for code readability and understanding which is one of the lodestar's objective (to stay readable and declarative)

if (blobSidecarsMinEpoch >= config.DENEB_FORK_EPOCH) {
const slotsToDelete = await db.blobSidecarsArchive.keys({lt: computeStartSlotAtEpoch(blobSidecarsMinEpoch)});
if (slotsToDelete.length > 0) {
await db.blobSidecarsArchive.batchDelete(slotsToDelete);
logger.verbose(
`blobSidecars prune: batchDelete range ${slotsToDelete[0]}..${slotsToDelete[slotsToDelete.length - 1]}`
);
} else {
logger.verbose(`blobSidecars prune: no entries before epoch ${blobSidecarsMinEpoch}`);
}
}
} else {
logger.verbose("blobSidecars pruning skipped: archiveBlobEpochs set to Infinity");
}
}

Expand Down
6 changes: 5 additions & 1 deletion packages/beacon-node/src/chain/archiver/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ const PROCESS_FINALIZED_CHECKPOINT_QUEUE_LEN = 256;

export type ArchiverOpts = StatesArchiverOpts & {
disableArchiveOnCheckpoint?: boolean;
archiveBlobEpochs?: number;
};

type ProposalStats = {
Expand All @@ -37,6 +38,7 @@ export class Archiver {

private prevFinalized: CheckpointWithHex;
private readonly statesArchiver: StatesArchiver;
private archiveBlobEpochs?: number;

constructor(
private readonly db: IBeaconDb,
Expand All @@ -45,6 +47,7 @@ export class Archiver {
signal: AbortSignal,
opts: ArchiverOpts
) {
this.archiveBlobEpochs = opts.archiveBlobEpochs;
this.statesArchiver = new StatesArchiver(chain.regen, db, logger, opts);
this.prevFinalized = chain.forkChoice.getFinalizedCheckpoint();
this.jobQueue = new JobItemQueue<[CheckpointWithHex], void>(this.processFinalizedCheckpoint, {
Expand Down Expand Up @@ -96,7 +99,8 @@ export class Archiver {
this.chain.lightClientServer,
this.logger,
finalized,
this.chain.clock.currentEpoch
this.chain.clock.currentEpoch,
this.archiveBlobEpochs
);
this.prevFinalized = finalized;

Expand Down
1 change: 1 addition & 0 deletions packages/beacon-node/src/chain/options.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ export type IChainOptions = BlockProcessOpts &
trustedSetup?: string;
broadcastValidationStrictness?: string;
minSameMessageSignatureSetsToBatch: number;
archiveBlobEpochs?: number;
};

export type BlockProcessOpts = {
Expand Down
8 changes: 8 additions & 0 deletions packages/cli/src/options/beaconNodeOptions/chain.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ export type ChainArgs = {
broadcastValidationStrictness?: string;
"chain.minSameMessageSignatureSetsToBatch"?: number;
"chain.maxShufflingCacheEpochs"?: number;
"chain.archiveBlobEpochs"?: number;
};

export function parseArgs(args: ChainArgs): IBeaconNodeOptions["chain"] {
Expand Down Expand Up @@ -53,6 +54,7 @@ export function parseArgs(args: ChainArgs): IBeaconNodeOptions["chain"] {
minSameMessageSignatureSetsToBatch:
args["chain.minSameMessageSignatureSetsToBatch"] ?? defaultOptions.chain.minSameMessageSignatureSetsToBatch,
maxShufflingCacheEpochs: args["chain.maxShufflingCacheEpochs"] ?? defaultOptions.chain.maxShufflingCacheEpochs,
archiveBlobEpochs: args["chain.archiveBlobEpochs"],
};
}

Expand Down Expand Up @@ -212,4 +214,10 @@ Will double processing times. Use only for debugging purposes.",
default: defaultOptions.chain.maxShufflingCacheEpochs,
group: "chain",
},

"chain.archiveBlobEpochs": {
description: "Number of epochs to retain finalized blobs (minimum of MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS)",
type: "number",
group: "chain",
},
};
2 changes: 2 additions & 0 deletions packages/cli/test/unit/options/beaconNodeOptions.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ describe("options / beaconNodeOptions", () => {
"chain.trustedSetup": "",
"chain.minSameMessageSignatureSetsToBatch": 32,
"chain.maxShufflingCacheEpochs": 100,
"chain.archiveBlobEpochs": 10000,
emitPayloadAttributes: false,

eth1: true,
Expand Down Expand Up @@ -139,6 +140,7 @@ describe("options / beaconNodeOptions", () => {
trustedSetup: "",
minSameMessageSignatureSetsToBatch: 32,
maxShufflingCacheEpochs: 100,
archiveBlobEpochs: 10000,
},
eth1: {
enabled: true,
Expand Down