Customizable genesis files

[uprendis]

Rework genesis file format and genesis processing:

• There may be multiple genesis files for the same network, featuring different starting data (varying by starting epoch and a degree of pruning). Archive genesis files will allow to start an archive node without processing all network transactions
• Genesis file consists of network header and enveloped zip container, which stores 3 self-hashed files (S) for blocks, epochs and EVM data. Unlike previous format, it allows to start reading data right away without a separate decoding phase
• (S) Self-hashed file (stream without random access) allows to read data from an untrusted source with a guarantee that either data will match the hash or an error will be raised. It requires to know the hash of the file in advance. It works similarly to torrent files, where file is separated into hashed pieces, thus doesn't require reading a file twice

Other changes:

• erase "sfc" API namespace
• allow deterministic state pruning with --prune.exact
• optimize EVM DB verification command
• allow non-zero addresses for internal txs

[hadv]

@uprendis Will we include below branch also into this PR?

https://github.com/uprendis/go-opera/commits/feature/add-fileshash-tests

[uprendis]

@uprendis Will we include below branch also into this PR?

https://github.com/uprendis/go-opera/commits/feature/add-fileshash-tests

Sure, I thought we'll add it after this PR. You can make a PR straight into this branch feature/customizable-genesis-file

[hadv]

@uprendis Will we include below branch also into this PR?
https://github.com/uprendis/go-opera/commits/feature/add-fileshash-tests

Sure, I thought we'll add it after this PR. You can make a PR straight into this branch feature/customizable-genesis-file

okay just thought you forgot to include it into this PR. If you have plan to add it later after this PR then it's fine.

[cyberbono3]

@uprendis what kind of manual testing scenarios should be performed to ensure your code works as expected?

[uprendis]

Here's the interface:
--genesis flag:
You can use to specify a genesis file of the new format. Previous format isn't supported

--genesis.allowExperimental:
Add this flag to run an unknown genesis file. There's a hardcoded list of known genesis files in cmd/opera/launcher/params.go , which currently contains genesis files of testnet: https://github.com/uprendis/lachesis_launch/blob/feature/add-snapsync-instruction/docs/genesis-files.md

opera --datadir /datadir/ export genesis output.g --export.evm.mode=ext-mpt
Generation of genesis file from current datadir
There's 3 modes for --export.evm.mode:

• none: doesn't export any EVM data
• mpt: exports only MPT data (used for pruned genesis files, like in the wiki table)
• ext-mpt: exports only MPT data and preimages (used for full genesis files, like in the wiki table)
  Ideally, genesis file should be generated in a beginning of an epoch. It'll work otherwise too, but will contain excessive data. You may make node pause in a beginning of an epoch either by hardcoding it, or by using this branch: uprendis:feature/extend-exitwhensynced with flag --exitwhensynced.epoch 6226

opera snapshot prune-state --prune.exact
Addition of new flag --prune.exact which makes it use a leveldb DB instead of a memory bloom filter for memorizing used MPT items. This way, it'll erase exactly all unused MPT items and the process will be deterministic. It's handy for generation of pruned genesis files

I may suggest such testcases:

• Try to sync up testnet from different genesis files using either fullsync or snapsync and verify that they return the same data via API (unless this data is pruned in genesis file)
• Try to generate custom genesis files and verify that it works as expected. Try to repeat generation of genesis files for epoch 6226 for testnet and verify that you get the exact same files

[cyberbono3]

lgtm. I suggest to add some testing since it is a big PR.