Explore better ways to work with ethereumjs-testing

[evertonfraga]

It is a somewhat heavy burden to work with the official tests files. As a consequence of its size, we've been treating it in a very special way. The package is kinda easily handled locally with local npm caches, but that's not the case with the CI.

With a total cache bundle surpassing 1.4GB, unpacking the cache can take even longer than installing dependencies directly. After some investigation, I found out that using git submodule update is generally much faster (48s) than unpacking the overall bundle (3 to 7 minutes).

This is now a problem, because I wanted to have an unified cached node_modules to load in all tests. Some ideas I had:

1) Link submodule from a deeper ethereum/tests directory
❌ Can't be done, as we use all of the submdule top level directories.

2) Let the VM repo manage the ethereum/tests submodule directly
❌ That would be some kind of shortcut, but it defeats the purpose of ethereumjs-testing versioning and releases.

3) Release ethereumjs-testing without the submodule

• The overall cache size for this repo would be 83% smaller.
• Each consumer would lazily load the test files from github.

For that to work, we'd need to:

• setup script to run the git submodule update when needed
• teardown script to delete these files by the end of the tests, so they don't reach the job cache

4) Load test files as a submodule on the vm

• each package that uses it should have their own test loading logic
• git submodule checkout is way faster than cache load
• node_modules cache size will be alleviated

2020-11-12 edit Added 4th item

[evertonfraga]

I appreciate the work done so far on this topic. Some code was moved to the VM (#881). But as we do the same with other packages (Blockchain, Tx), there'll be an increasing need to reuse the code (getTests() functions and alike) in the other packages.

How about we keep the getTests() code in the ethereumjs-testing, and remove the submodule from the library, adding it to the library consumer? In the monorepo case, it could live in the root directory, and each package that uses it would pass the path to the tests.

This work, if agreed, can be done as a preparatory step to the Yarn workspaces (#900) PR.

[holgerd77]

@ev These things were already considered when this code was moved to the VM (I just can't find the reference any more), so I would vote against here. There is actually no ethereumjs-testing integration in Blockchain as far as my search was successful 😄 , so there is only the tx library. Most of the getTests options are very much VM specific and the Tx test runner is using a very simple flavor. This should be easily replaceable by some 5-10 LOC extract from the current getTests logic or directly with some simpler and more modern file access approach. At the same time it is anticipated that the VM test execution might benefit substantially from the current tight integration by allowing to more easily experiment with alternative file access patterns and/or also using more modern file access approaches.

[holgerd77]

Closed by #953