post-genesis transition #426

gballet · 2024-05-02T13:41:16Z

This is a squashed + rebased version of #304. It adds the ability:

To undergo the verkle transition "verge" past the genesis block, which so far was impossible with kaustinen-with-shapella.
To perform a shadow fork of holesky in the special kurtosis setup defined by the devops team.

It is also serving as the basis for multiple other branches, to be merged at a later time:

The branch to replay historical blocks in verkle mode
The branch to execute transition tests

quell linter issue support Transition post tree in conversion Refactor transition post genesis (#311) * rewrite per-block conversion pointer management * remove unused method * fix: a branch that can verge at genesis or post genesis (#314) * fix: import cycle in conversion refactor (#315) * fix shadowfork panic in OpenStorageTrie fix OpenStorageTrie: return an error instead of panicking if the account tree not a verkle tree (#321) add a switch to force proof in blocks (#322) * add a switch to force proof in blocks * activate switch * fix switch type add switch to override the stride of the overlay conversion (#323) * add switch to override the stride of the overlay conversion * set a default stride of 10k add a few traces for relaunch add missing step in the chain of setting the override for overlay stride fix: save and load transition state for block processing (#324) * fix: save and load transition state for block processing * log whether the tree is verkle in LoadTransitionState * fix: ensure the transition is marked as started in insertChain * dump saved address * fix nil pointer panic * remove stacktrace that is no longer useful * fix a panic * fix build * check: copy current account address BEFORE it's saved * mandatory panic fix * Remove debug fmt.Println * more cleanup + comments fix: ensure StorageProcessed is properly recovered add traces for gas pool overflow cmd/{geth, utils}: add a command to clear verkle costs (#326) * cmd/{geth, utils}: add a command to clear verkle costs fix: boolean issue fix: load finalization state in FinalizeAndAssemble (#340) * Conversion and TransitionTrie fixes (#346) * fixes Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * remove old comment Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> --------- Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * trace cleanup --------- Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com> fix: check for nil override in NewBlockChain (#357) fix rebase issue fix a few rebase issues add debug traces fix: assume that having to create a new transaction state mean it hasn't happened. This is a workaround, because it will always be false when restarting a node that has started the transition but not necessarily completed it. add logs to debug shadowfork persist conversion state to db and use an LRU cache for active transition states (#375) * persist conversion state to db * fix: don't recreate LRU when writing state * opt: only write state to db if not already present in LRU * fix: rlp can't encode TransitionState * fix: use gob because binary.Write does not support structs 🤦‍♂️ * fix: nil pointer panic * add logs to debug shadowfork * no such thing as not enough traces * ditto * fix stupid bug * add a comment for readability * add more traces * Lock the state transition during conversion (#384) * heavy handed approach: lock the state transition during conversion * also lock transition state loading/unloading * reduce logs verbosity * add conversion test to workflow (#386) * add conversion test to workflow * mandatory -f switch fix in rm * forgot & at the end of the geth command * remove incorrect kill * add debug traces * add an overlay stride * fix typo * Apply suggestions from code review eth: add debug_conversionStatus RPC call (#392) * eth: add debug_conversionStatus RPC call * add debug trace4s * Apply suggestions from code review * export started/ended fields fix post-verge sync (#404) * fix post-verge sync * review: fix truncated comment

core/block_validator.go

jsign · 2024-05-02T13:50:20Z

core/blockchain.go

+		// TODO this only works when resuming a chain that has already gone
+		// through the conversion. All pointers should be saved to the DB
+		// for it to be able to recover if interrupted during the transition
+		// but that's left out to a later PR since there's not really a need
+		// right now.
+		bc.stateCache.InitTransitionStatus(true, true)


I think this has caused some confusions and problems before, but we'll see.

it works for all tests that I'm aware of (shadowfork, post-genesis transition and kaustinen sync), but I agree it's tacky. There is a TODO however, so this will be fixed eventually.

core/overlay/conversion.go

jsign · 2024-05-02T13:57:05Z

core/state/statedb.go

-			if err := s.snaps.Cap(root, 128); err != nil {
+			if err := s.snaps.Cap(root, 8192); err != nil {


Nit: I think there was a const for this?

there is, but it's defined in a different package. I will create a separate PR to fix this at the geth level.

core/state_transition.go

cmd/utils/flags.go

core/blockchain.go

consensus/beacon/consensus.go

core/block_validator.go

Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>

core/blockchain.go

Support post-genesis transition Reactivate a post-genesis transition for a kaustinen-like testnet that goes through the conversion. It adds the ability: * To undergo the verkle transition "verge" past the genesis block, which so far was impossible with kaustinen-with-shapella. * To perform a shadow fork of holesky in the special kurtosis setup defined by the devops team. It is also serving as the basis for multiple other branches, to be merged at a later time: * The branch to replay historical blocks in verkle * The branch to execute transition tests Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign approved these changes May 2, 2024

View reviewed changes

gballet marked this pull request as ready for review May 3, 2024 07:24

gballet requested a review from holiman as a code owner May 3, 2024 07:24