forest-cli archive export --epoch=X

[lemmih]

A Forest node can generate up-to-date snapshot files. However, we often want to take large snapshots and trim them to a specific epoch. This can be done entirely without network access.

The forest-cli archive export command takes a snapshot file as input and yields a smaller snapshot file as output. For example, the input snapshot might cover epochs 1 to 5000, and the output snapshot could cover epochs 1000 to 2000.

Subtasks:

• Take CAR file as input,
• Index CAR file using car-backed Blockstore,
• Get recent-roots and epoch from cli flags,
• Use export function to generate snapshot data,
• Compress with zst.

Error handling:

• recent-roots too low (less than chain_finality),
• recent-roots too high (greater than available roots),
• epoch not in CAR input file,
• input file may be malformed or corrupted.

Export function

forest/src/chain/store/chain_store.rs

Lines 528 to 535 in 4f0d75f

	pub async fn export<W, D>(
	&self,
	tipset: &Tipset,
	recent_roots: ChainEpoch,
	writer: W,
	compressed: bool,
	skip_checksum: bool,
	) -> Result<Option<digest::Output<D>>, Error>

[lemmih]

Testing can be done by downloading a calibnet snapshot and re-exporting it in various ways (different offsets, different number of tipsets).

[lemmih]

Automated testing with the MemoryDB would also be nice.

[ruseinov]

@lemmih Do we also want the checksum here?

[lemmih]

@lemmih Do we also want the checksum here?

We can leave out the checksum for now.

[ruseinov]

One more question: do we really want to call a param recent-roots? I would probably do something like
epoch-from, epoch-to and make sure that epoch-from is at least chain_finality` (defaulting to it too).

We could come up with even better naming perhaps
--epoch and --depth.

[lemmih]

One more question: do we really want to call a param recent-roots? I would probably do something like epoch-from, epoch-to and make sure that epoch-from is at least chain_finality` (defaulting to it too).

We could come up with even better naming perhaps --epoch and --depth.

--epoch and --depth looks good.