Make Warp/Light sync and custom sync protocols work by design rather than by accident

[nazar-pc]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Motivation

Unless using SyncMode::LightState or SyncMode::Warp when attempting to insert a block significantly higher than the tip it will fail here:

polkadot-sdk/substrate/client/state-db/src/noncanonical.rs

Lines 263 to 274 in 05a8ba6

	} else if self.last_canonicalized.is_some() {
	if number < front_block_number || number > front_block_number + self.levels.len() as u64
	{
	trace!(
	target: LOG_TARGET,
	"Failed to insert block {}, current is {} .. {})",
	number,
	front_block_number,
	front_block_number + self.levels.len() as u64,
	);
	return Err(StateDbError::InvalidBlockNumber)
	}

The reason it works in those cases is because of this awkwardness:

polkadot-sdk/substrate/client/service/src/config.rs

Lines 241 to 245 in 05a8ba6

	/// Returns true if the genesis state writing will be skipped while initializing the genesis
	/// block.
	pub fn no_genesis(&self) -> bool {
	matches!(self.network.sync_mode, SyncMode::LightState { .. } | SyncMode::Warp { .. })
	}

Which essentially says we will generate, but not store genesis state on disk, which in turns allows to bypass previous snippet because suddenly when the first block is inserted self.last_canonicalized.is_some() == false.

This whole thing if basically working by accident as far as I'm concerned. There is no fundamental reason why it should be this way and in fact I'd like to have a sync mechanism that can make those jumps at a later time as well, not just after genesis.

Imagine chain has 10M blocks and current database has 1M blocks only, why shouldn't it be possible to Warp sync close to 10M in this case without wiping the whole database first?

This is one of the issues leading to #4407

cc @shamil-gadelshin

Request

I'm not sure what I'm requesting exactly, but I'm looking for some refactoring that would decouple these things and ideally make it possible to insert blocks much higher than current tip if there is a need for that in the protocol.

Solution

Could be as simple as another boolean to bypass mentioned check, but I'm not sure about expected invariants, so things might be more complex than that. Open for suggestions.

Are you willing to help with this request?

Yes!