Skip to content

Commit

Permalink
Verkle tree-based state with overlay transition
Browse files Browse the repository at this point in the history 
Squash the main verkle PR ahead of rebase

don't call Bytes() in GetTreeKey (ethereum#137)

trie: avoid endianness conversion in GetTreeKey (ethereum#140)

* trie/utils: add concrete expected value in trie key generation test

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* mod: update to latest go-verkle

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* trie/utils: avoid endianness conversions

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* apply review changes & update to official go-verkle version

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

upgrade go-verkle to CoW version and get TestProcessVerkle to build (ethereum#138)

updating ci to use self-hosted machine (ethereum#143)

fix: storage offset in non-header group + reuse of value buffer (ethereum#145)

dedup call to ChunkifyCode, same as replay branch (ethereum#156)

* dedup call to ChunkifyCode, same as replay branch

* fix some linter issues

fix code offset in tree update (ethereum#157)

fix REVERT in state processor test execution (ethereum#158)

* fix code offset in tree update

* fix REVERT in test execution

save on key hashing: lump code size update with first code chunk group (ethereum#159)

fix code chunk key calculation and storage key calculation (ethereum#161)

* fix codeKey calculation

* Remove

* fix storageOffset

* fix the fix to the fix to the offset fix

* Remove copy/pasted, unused code in test

* fix linter

---------

Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>

fix: infinite loop when calling extcodecopy on empty code (ethereum#151)

upgrade to latest go-verkle

fix: only update code in the tree if it's dirty (ethereum#174)

fix: read-touch the code size and Keccak of the origin (ethereum#175)

List of changes for converting a sepolia database (ethereum#182)

* naive conversion rebased on top of beverly hills

* changes for the sepolia shadow fork conversion

* fixes to please the linter

* fixes to please the linter

Unified point cache (ethereum#180)

* Unified point cache

* Use cache for Try*Account

* alter Trie interface to use caching for slots (ethereum#181)

* alter Trie interface to use caching for slots

* fix: use a lock to protect the point cache (ethereum#185)

* use fastest non-master go-verkle version & pull trie/Verkle.go changes to use new api (ethereum#184)

* mod: update to fastest go-verkle version today

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* trie/verkle: use new batch serialization api

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

---------

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

---------

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>

* fix: TryDelete signature in unit tests

---------

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>

trie/utils: fix potential overflow (ethereum#191)

* trie/utils: fix potential overflow

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* trie/utils: receive storage key as a byte slice

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* revert formatter changes

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* trie/utils: fix mod 256

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

---------

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

trie/utils: fix incorrect bigint assignment (ethereum#193)

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

upgrade precomp link to fix CI

fix: add missing code size&keccak leaves in empty accounts (ethereum#192)

fixes to use the latest go-verkle@master (ethereum#197)

* fixes to use the latest go-verkle@master

* linter fixes

* linter fixes for tests

* fix: use jsign's go-verkle fix

refactor: remove unused (*StateDB).GetXLittleEndian methods (ethereum#204)

fix gas accounting issue in state_processor_test.go (ethereum#207)

update go-verkle not to use StatelessNode anymore (ethereum#206)

* update go-verkle not to use StatelessNode anymore

* update go-verkle to latest

refactor: move verkle gas accounting to its own block in TransitionDB (ethereum#208)

fix a panic in deserializeVerkleProof if GetProofItems returns a nil ProofElements

use the cachingDB instead of a custom VerkleDB (ethereum#209)

* use the cachingDB instead of a custom VerkleDB

* fix stack trace in LES

remove holiman from CODEOWNERS as he gets too many emails

read from tree in state object if the snapshot is nil (ethereum#205)

add missing error checks for the root node type (ethereum#214)

implement OpenStorageTrie for verkle trees (ethereum#210)

* implement OpenStorageTrie for verkle trees

* add a few comments for future maintenance

* fix linter issue

fix: copy balance leaf to new buffer in TryGetAccount (ethereum#217)

implement some heretofore unimplemented iterator methods (ethereum#219)

params: move verkle params to their own file (ethereum#228)

fix: proper number of chunk evals (ethereum#215)

overlay transition (ethereum#244)

* overlay transition

Fix some bugs identified in the code review

Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>

Include base -> overlay key-values migration logic (ethereum#199)

* mod: add go-verkle version with key-value migration new apis

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* core/stateprocessor: use constant for max number of migrated key-values

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* core: add base->overlay key-values migration logic

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* core: fix some compiler errors

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* trie: consider removing transition trie api in the future

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* mod: use latest go-verkle

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

---------

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

fix some unit tests errors

get convresion block from file

fix compilation issues

fix initialization issue in migrator

fix: changes needed to run the first 28 blocks

important sutff: fix the banner

fix: use nonce instead of balance in nonce leaf (ethereum#202)

fixes for performing the overlay transition (ethereum#203)

* fixes for performing the overlay transition

* fixes for the full replay

* fix: deletion-and-recreation of EoA

* fixes to replay 2M+ blocks

* upgrade to go-verkle@master

* fix: proper number of chunk evals

* rewrite conversion loop to fix known issues

changes to make replay work with the overlay method (ethereum#216)

* fixes for performing the overlay transition

fixes for the full replay

fix: deletion-and-recreation of EoA

fixes to replay 2M+ blocks

upgrade to go-verkle@master

fix: proper number of chunk evals

rewrite conversion loop to fix known issues

changes to make replay work with the overlay method

fixes to replay 2M+ blocks

update to latest go-verkle@master

* use a PBSS-like scheme for internal nodes (ethereum#221)

* use a PBSS-like scheme for internal nodes

* a couple of fixes coming from debugging replay

* fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (ethereum#222)

* fixes for pbss replay (ethereum#227)

* fixes for pbss replay

* trie/verkle: use capped batch size (ethereum#229)

* trie/verkle: use capped batch size

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* trie/verkle: avoid path variable allocation per db.Put

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* don't keep more than 32 state root conversions in RAM (ethereum#230)

---------

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>

* cleanup some code

* mod: update go-verkle

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* re-enable snapshot (ethereum#231)

* re-enable cancun block / snapshot (ethereum#226)

* clear storage conversion key upon translating account (ethereum#234)

* clear storage conversion key upon translating account

* mod: use latest go-verkle

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

---------

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>

* fix: self-deadlock with translated root map mutex (ethereum#236)

* return compressed commitment as root commitment (ethereum#237)

---------

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>

---------

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>

---------

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>

fix first panic in *TransitionTrie.Copy()

upgrade go-verkle to latest master

mod: update go-verkle (ethereum#239)

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

core: print state root every 100 blocks (ethereum#240)

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

fix: only Commit the account trie (ethereum#242)

fixes to get TestProcessVerkle to work with the overlay branch (ethereum#238)

* fixes to get TestProcessVerkle to work with the overlay branch

* fix all panics in verkle state processor test

* fix proof verification

move transition management to cachingDB

* fix: mark the verkle transition as started if it's ended without being started

* fix the verkle state processing test

* fix linter errors

* Add a function to clear verkle params for replay

* fix: handle TransitionTrie in OpenStorageTrie

* fix linter issue

* fix the deleted account error (ethereum#247)

* code cleanup (ethereum#248)

* fix: don't error on a missing conversion.txt (ethereum#249)

* Overlay Tree preimages exporting and usage (ethereum#246)

* export overlay preimages tool

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* use preimages flat file in overlay tree migration logic

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* cmd/geth: add --roothash to overlay tree preimage exporting command

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* cleanup

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* review feedback

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

---------

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* fix: reduce the PR footprint (ethereum#250)

* fix: don't fail when preimages.bin is missing (ethereum#251)

* fix: don't fail when preimages.bin is missing

* fix: don't open the preimages file when outside of transition

---------

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>

review changes

remove replay-specific code
  • Loading branch information
@gballet @rjl493456442
gballet authored and rjl493456442 committed Aug 28, 2023
1 parent df64a21 commit f1ae3e8
Show file tree
Hide file tree
Showing 57 changed files with 3,963 additions and 58 deletions.
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
View file
Validating CODEOWNERS rules …
Expand Up @@ -6,7 +6,7 @@ accounts/scwallet @gballet
accounts/abi @gballet @MariusVanDerWijden
cmd/clef @holiman
consensus @karalabe
core/ @karalabe @holiman @rjl493456442
core/ @karalabe @rjl493456442
eth/ @karalabe @holiman @rjl493456442
eth/catalyst/ @gballet
eth/tracers/ @s1na
Expand Down
48 changes: 48 additions & 0 deletions .github/workflows/go.yml
View file
@@ -0,0 +1,48 @@
name: Go lint and test

on:
push:
branches: [ master ]
pull_request:
branches: [ master, verkle-trie-proof-in-block-rebased, verkle-trie-post-merge, beverly-hills-head, 'verkle/replay-change-with-tree-group-tryupdate' ]
workflow_dispatch:

jobs:
build:
runs-on: self-hosted
steps:
- uses: actions/checkout@v2
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: 1.18
- name: Build
run: go build -v ./...

lint:
runs-on: self-hosted
steps:
- uses: actions/checkout@v2
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: 1.18
- name: Download golangci-lint
run: wget -O- -nv https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s latest
- name: Lint
run: ./bin/golangci-lint run
- name: Vet
run: go vet

test:
runs-on: self-hosted
steps:
- uses: actions/checkout@v2
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: 1.18
- name: Download precomputed points
run: wget -nv https://github.com/gballet/go-verkle/releases/download/banderwagonv3/precomp -Otrie/utils/precomp
- name: Test
run: go test ./...
38 changes: 38 additions & 0 deletions cmd/geth/chaincmd.go
View file
Expand Up @@ -150,6 +150,17 @@ It's deprecated, please use "geth db import" instead.
Description: `
The export-preimages command exports hash preimages to an RLP encoded stream.
It's deprecated, please use "geth db export" instead.
`,
}
exportOverlayPreimagesCommand = &cli.Command{
Action: exportOverlayPreimages,
Name: "export-overlay-preimages",
Usage: "Export the preimage in overlay tree migration order",
ArgsUsage: "<dumpfile>",
Flags: flags.Merge([]cli.Flag{utils.TreeRootFlag}, utils.DatabasePathFlags),
Description: `
The export-overlay-preimages command exports hash preimages to a flat file, in exactly
the expected order for the overlay tree migration.
`,
}
dumpCommand = &cli.Command{
Expand Down Expand Up @@ -407,6 +418,33 @@ func exportPreimages(ctx *cli.Context) error {
return nil
}

// exportOverlayPreimages dumps the preimage data to a flat file.
func exportOverlayPreimages(ctx *cli.Context) error {
if ctx.Args().Len() < 1 {
utils.Fatalf("This command requires an argument.")
}
stack, _ := makeConfigNode(ctx)
defer stack.Close()

chain, _ := utils.MakeChain(ctx, stack, true)

var root common.Hash
if ctx.String(utils.TreeRootFlag.Name) != "" {
rootBytes := common.FromHex(ctx.String(utils.StartKeyFlag.Name))
if len(rootBytes) != common.HashLength {
return fmt.Errorf("invalid root hash length")
}
root = common.BytesToHash(rootBytes)
}

start := time.Now()
if err := utils.ExportOverlayPreimages(chain, ctx.Args().First(), root); err != nil {
utils.Fatalf("Export error: %v\n", err)
}
fmt.Printf("Export done in %v\n", time.Since(start))
return nil
}

func parseDumpConfig(ctx *cli.Context, stack *node.Node) (*state.DumpConfig, ethdb.Database, common.Hash, error) {
db := utils.MakeChainDatabase(ctx, stack, true)
var header *types.Header
Expand Down
1 change: 1 addition & 0 deletions cmd/geth/main.go
View file
Expand Up @@ -210,6 +210,7 @@ func init() {
exportCommand,
importPreimagesCommand,
exportPreimagesCommand,
exportOverlayPreimagesCommand,
removedbCommand,
dumpCommand,
dumpGenesisCommand,
Expand Down

0 comments on commit f1ae3e8

Please sign in to comment.