VM v5 Release Planning

[holgerd77]

Now with the monorepo transition being in its latests phases we can start on planning the next major VM release, various things are on the door for this. I'll try to collect here what I can think of and we can further discuss, if we have some agreement on further additions please just add to the list below:

TODO List

• VM: Init method to prevent race conditions PR Add initialization method to prevent race conditions #665 by @rumkin, review/merge , @evertonfraga
• VM: Opcode refactor PR Refactor opcodes #664 by @rumkin, review/merge, @cgewecke, @alcuadrado
• VM: Integrate Block v3.0.0 and new Blockchain versions, @evertonfraga
  • Block library TypeScript v3.0.0 (see New release v3.0.0 ethereumjs-block#74, work already done, just needs release PR + npm publishing), PR Block release v3.0.0 #682, @evertonfraga
  • Blockchain: Integrate Block v3.0.0, promisified Blockchain version Promisify blockchain and ethash packages #833 , @evertonfraga @ryanio
• VM: Integrate MPT release v4.0.0, @ryanio, PR Update merkle-patricia-tree to v4, move account trie-related methods to StateManager #787
  • Block: integrate into Block library - eventually along v3.0.0 Block release v3.0.0 #682, @evertonfraga
  • Account: eventually integrate into Account library (devDependencies for tests), @ryanio, update: dependency removed, PR Update merkle-patricia-tree to v4, move account trie-related methods to StateManager #787
• VM: Extract the state manager to have its own package, see section in VM README for context, @ryanio, update: decided (missing reference, can't find) to keep it integrated for now
  • Integrate promises into StateManager, Integrate promises into StateManager #719
  • Add Interface for StateManager, StateManager interface #763
• All: Update ethereumjs-util to v7.0.2, Update ethereumjs-util to v7 #748 (double-check if all occurences updated!)
• All: Update to a a new ethereumjs-config version integrating e.g. Upgrade primary build target to ES2017, add browser build ethereumjs-config#28 (ES2017 build target, separate browser build)
• Analyze and fix speed deterioration of MPT v4 library, see Analyze speed deterioration of v4 merkle-patricia-tree#124, @ryanio
  • Add simple benchmarking tool to VM, PR VM: add simple benchmarking tool for mainnet blocks #794, @s1na, @ryanio, @holgerd77
  • Run benchmarks on pre-deteriorated, post-deteriorated and fixed MPT and compare
  • Compare with MPTs own benchmarks and see if consistent
• Finish all Hardfork backports for VM, @jochem-brouwer
  • SpuriousDragon HF support, SpuriousDragon HF Support #791, @holgerd77
  • TangerineWhistle HF support, TangerineWhistle HF Support #807, @holgerd77, @jochem-brouwer
  • Finalize and review Homestead support PR, [VM] fix blockchain tests and add Homestead support #815, @jochem-brouwer, all
  • Finalize and review chainstart (Frontier) support PR, Add frontier support #828, @jochem-brouwer, all
  • Optional but nice (shouldn't bee too much extra work and then we are really complete): add DAO HF, all
• Add Berlin HF EIPs
  • Finalize and review BLS precompile EIP PR, VM: Implement EIP 2537 (BLS12-381 precompiles) #785, @jochem, all -> Postponed/skipped on release due to EIP HF integration uncertainties due to EVM384 alternative proposal
  • Review, eventually finalize subroutine EIP PR, Implement EIP-2315: subroutines #754, @cgewecke, all
• Set VM default HF to Istanbul (MuirGlacier is problematic since not available on all testnets (e.g. Rinkeby which would lead to unexpected behavior if people just change the network without further thinking about this (which is likely))

Be Happy! 🦄 😄 🌻

Discussion

• VM: Do we finally want to expose the reworked Interpreter API Sina @s1na introduced along the v4.0.0 release ( see "EVM Modularization and Structural Refactoring" section as well as linked PR within the section)? Related: VM: Run transaction with custom EVM/Interpreter / API Discussion #636
  • Update: agreement that there is currently still too much movement around this, so won't be integrated in v5 but rather postponed to a subsequent major version release
• VM, merkle-patricia-tree: Do we want to also wait for the TypeScript / Promisification v4.0.0 release of the merkle tree library to add here or do we want to give this a bit more time (this would also need an update on the Block and eventually the Account (only dev dependency) library)? @ryanio
  • -> Update: MPT release is imminent, so this can be integrated, have moved to TODO

Notes

1. Ownership: I've assigned team members already involved or I thought fitting to the tasks, if you can't/don't want to take over just replace with OPEN or something.

2. Date: Just as some orientation, I would roughly target a date within a range of 2-3 weeks for a possible release, let's take April 22, Wednesday as a starter, let me know if you have comments on that

Anything else? Something missing? Other suggestions?

[holgerd77]

Just some notes, before I forget, will work this better into the description above and generally update the issue list early next week.

• Work here turns out to be likely to extensive to match the April 22 date in a couple of days, we'll likely have to move here by at least I would say 2 weeks
• MPT promisification PR by @ryanio is merged (yeah 🎉, great work) and a v4 release is relatively imminent Release - v4.0.0 merkle-patricia-tree#111, so we can likely take a MPT update in
• @cgewecke has committed to do some work on the ethereumjs-util library and finalize Update secp2561 ECDSA dependency to v4.0.0 ethereumjs-util#228, wonder if we should also take this in (would be also a major release on the util library) since this would significantly improve the installation process, which is currently really a pain regarding the crypto packages and source for many complaints. Not sure, Chris will you be able to work on this relatively soon (next week optimally?)?

We generally have to look at the dependency tree (by @evertonfraga) really closely regarding all these things to not miss some updates.

[cgewecke]

Chris will you be able to work on this relatively soon (next week optimally?)?

@holgerd77 Yes, I will start working on this Monday, April 20.

[kumavis]

some ideas from earlier days of ethereumjs-*

• merkle-patricia-tree if you make getting the current state root of a merkle-patricia-tree an async op, you can put off ALL hash calculations for new values until the root is read. if the root is never read, you saved yourself a bunch of work
• vm if you make the storage backend easily pluggable, you can run the vm off of rpc requests
  https://github.com/ethereumjs/ethereumjs-vm/blob/0f10fcfdbc55ceadbff6adeb54bca6df393ef62f/lib/hooked.js#L77-L83 This is hugely useful for anyone who wants to run the vm while accessing state remotely. sad to see support for it was dropped.

basicaclly scifi:

• vm: out-of-order execution. (not speculative execution or branch prediction) a significant change, but if you make all values you push onto the stack to be promises, you get great perf wins like parallel storage reads (huge for zero and light clients), and you let the js engine figure out the causality requirements. to really do it correctly you need to build a funky mutex machine around memory read/writes, but it sounds like a fun challenge. I started playing with this over the summer and you can make iterative progress while maintaining passing tests.
  e13503b...dynamic-execution#diff-f42577b259c8466289ed92b49e2a212c

[kumavis]

worth noting that ganache's forked blockchain feature also uses a json rpc client for state reads, this is how they are hooking in https://github.com/trufflesuite/ganache-core/blob/a5f43a4d666c82bce9da51f9ce1d15b6561e24f3/lib/utils/forkedblockchain.js#L89-L93

this usecase should be supported without having to override underscored properties/methods

[s1na]

Wrote down some thoughts re making EVM (Interpreter and EEI) public here. Probably makes sense to keep them private for this release.

[holgerd77]

For notice: have substantially updated the initial TODO list here to keep track with our latest discussions around various questions.

[holgerd77]

@evertonfraga Can you please give the TODO list from above an update, so that things won't get forgotten?

One additional thing which comes to my mind: we should do a last

npm ls ethereumjs-util

check on the final VM version to ensure that we haven't forgotten anything along the Uti v7 update.

Also: @ryanio what is the state of the MPT v4 update? Can we still manage to get this in?

[evertonfraga]

@holgerd77 sure. I will change the todo descriptions, if you don’t mind.

[ryanio]

@holgerd77 yes let's release mpt v4 on npm and i will finish my branch to include :)

[alcuadrado]

One additional thing which comes to my mind: we should do a last

npm ls ethereumjs-util

check on the final VM version to ensure that we haven't forgotten anything along the Uti v7 update.

I think this should be checked by a linter/script on the CI. I have a script to do this in Buidler, I hadn't proposed importing it here before, because it works with a monorepo. I'll open a PR now.

[evertonfraga]

Thanks @alcuadrado!

[alcuadrado]

Can we release and use @ethereumjs/config-typescript before this version gets published?

[evertonfraga]

Sure @alcuadrado, that’s on my plan. To make things more clear, tomorrow I’ll start the day creating a milestone with all the issues for the v5. You all will be welcome to suggest as well.

[jochem-brouwer]

Some thoughts on what I think should be included in V5 (as addition to things already mentioned here):

• Berlin HF support
• Support for all older HFs (we could skip DAO for now though)
• Remove callbacks from blockchain (blockchain is the only package which is now using callbacks as opposed to all other packages having replaced the callbacks by async functions).

Besides those, it might be a good idea to go over all issues and all currently open PRs in order to see if there are any breaking changes which we'd like to include in the V5 release.

[holgerd77]

Have updated the TODO list from above so that we can finally close in on a release, please add if something is missing.

[alcuadrado]

How is the Berlin support gonna be released? I think it should be somehow marked as experimental, with no guarantees of stability, as Berlin may include a different set of eips. Did we have some experimental apis before?

Just curious, I don't think this requieres too much thinking, just not making it default and a big warning.

[holgerd77]

@alcuadrado Yes, a bit as you described. Analogous to the previous forks we won't switch the default fork yet and mark the HF support as experimental. Once this has finalized we will likely release a new minor version with the stable-marked HF (still not made default). This worked pretty well in the past, no complains there at any time.

[alcuadrado]

Oh, that's nice. I thought the previous HFs were released after they were settled/activated.

[holgerd77]

Closed in favor of #907