Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/evm: make batched state-test execution possible #27318

Merged
merged 2 commits into from
May 23, 2023

Conversation

holiman
Copy link
Contributor

@holiman holiman commented May 22, 2023

This PR implements a feature that is already present in both nethermind and besu: the ability to run several state-tests in one instance.
For state-tests that are reasonably small, this makes a huge difference. Here's 1000 executions of a state-test which has a run-length of merely 28 steps.

Doing 1000 execs old style

real	0m55.600s
user	0m41.665s
sys	0m17.886s

Doing 1000 execs new style

real	0m1.400s
user	0m1.443s
sys	0m0.116s

Nearly one minute versus <2seconds. When running it in a fuzzing-setting, the speedups is less marked, somewhere in the 1x-2x region.
If the kzg stuff now has a static overhead, as suggested in #27258, this PR would definitely help (the fuzzer). Perhaps something similar can be done for evm t8n too.

myfunc() {
 for i in {1..1000}; do
        ./evm_master --json statetest /home/user/go/src/github.com/holiman/goevmlab/evms/testdata/cases/00000936-mixed-1.json  2>/dev/null 1>/dev/null
 done
}

myfunc2() {
 touch /tmp/foo
 rm /tmp/foo
 for i in {1..1000}; do
        echo "/home/user/go/src/github.com/holiman/goevmlab/evms/testdata/cases/00000936-mixed-1.json" >> /tmp/foo
 done
  echo "" >> /tmp/foo
  cat /tmp/foo | ./evm_batch --json statetest 2>/dev/null 1>/dev/null
}


echo "Doing 1000 execs old style"
time myfunc
echo "Doing 1000 execs new style"
time myfunc2

Copy link
Member

@MariusVanDerWijden MariusVanDerWijden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM a great feature which will speed up the fuzzing significantly

@holiman holiman added this to the 1.11.7 milestone May 23, 2023
@holiman holiman merged commit 1a18283 into ethereum:master May 23, 2023
1 of 2 checks passed
AskAlexSharov pushed a commit to ledgerwatch/erigon that referenced this pull request May 29, 2023
This implements batched state-test exectution, similar to
ethereum/go-ethereum#27318 .
Some speedtests, executing a state-test twice on current master takes
~4-5 seconds, and scales linerarly.
```
Doing 2 execs old style

real    0m8.185s
user    0m8.081s
sys     0m0.110s
```
Doing `100` executions on this PR -- a few seconds of ramp-up time, but
very quick execution after that :
```
Doing 100 execs v2

real    0m5.009s
user    0m4.560s
sys     0m0.508s
```
I also tested a version where I moved the db instantiation into the top
callsite, with the `MustOpen` and `.Close` only performed once, instead
of `100` times -- however, I noticed no additional speed gains from
doing so (my branch `batched_evm_v2`).

Therefore, I suspect that the slowdowns comes not from the db, but the
kzg library initialization.
antonydenyer pushed a commit to antonydenyer/go-ethereum that referenced this pull request Jul 28, 2023
implements the ability to run several state-tests in one instance. By not providing a statetest path to the `evm statetest` command, the path(s) will instead be read from `stdin`.
MoonShiesty pushed a commit to MoonShiesty/go-ethereum that referenced this pull request Aug 30, 2023
implements the ability to run several state-tests in one instance. By not providing a statetest path to the `evm statetest` command, the path(s) will instead be read from `stdin`.
@holiman holiman deleted the batched_exec branch October 11, 2023 07:27
devopsbo3 pushed a commit to HorizenOfficial/go-ethereum that referenced this pull request Nov 10, 2023
implements the ability to run several state-tests in one instance. By not providing a statetest path to the `evm statetest` command, the path(s) will instead be read from `stdin`.
devopsbo3 added a commit to HorizenOfficial/go-ethereum that referenced this pull request Nov 10, 2023
devopsbo3 added a commit to HorizenOfficial/go-ethereum that referenced this pull request Nov 10, 2023
devopsbo3 added a commit to HorizenOfficial/go-ethereum that referenced this pull request Nov 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants