Wasmi's execution speed is significantly slower than other runtime tools

[luxinyi0105]

Description

Wasmi's execution speed is significantly slower than other runtime tools while executing the given testcase

Versions and Environment

Tools version: wasmi_cli 0.31.0
Operating system: Ubuntu 22.04.1
Architecture: x86_64

Explanation of TestCase

The given testcase was simply mutated a wasm file, which was obtained by compiling C program generated with Csmith using Emscripten Compiler(Emcc).

The original C program is c_file.c, the compilation results with Emscripten is wasm_file.wasm, and its wat format is wat_file.wat.

We mutated the wat file to change its i32.sub in line 6638 with i32.div_s. The result after mutation is mutated_file.wat, and its wasm format is mutated_file.wasm.

Extra Info

Drawing from the experience of #751 and #750, I used the latest version 0.31.0 of wasmi and installed it via cargo build --profile bench in order to apply needed optimizations. The execution time of wasmi and other runtime tools are as follows:

wasmer： 10.325 s
wasmtime： 7.495 s
wasmedge with AOT mode： 6.010 s
wasm-micro-runtime： 1 m 9.837 s
wasm3： 5 m 52.228 s
wasmi： 39m33.818s

[Robbepop]

Thanks @luxinyi0105 for the detailed report! If Wasm3 already takes 5 minutes then wasmi's 40 minutes are still a lot but at least minimally realistic given that wasmi usually is roughly 3 times slower compared to Wasm3.

Also, what WAMR engine did you use? If it was the JIT engine I wonder why WAMR is so slow compared to the other JIT based Wasm engines.

Generally i32.div_s is a slow operation since it can error in several different ways that need to be caught. Optimizing Wasm engines can probably counter this by optimizing it to a certain extend which could explain these significant slowdowns.

[luxinyi0105]

Generally i32.div_s is a slow operation since it can error in several different ways that need to be caught. Optimizing Wasm engines can probably counter this by optimizing it to a certain extend which could explain these significant slowdowns.

Thanks for your reply. For WAMR, I compile and use the tool by default method, so perhaps the engine I use is "Interpreted".

Considering the execution time of wasmi, wasm3 and wamr, is the 40 minutes spent on wasmi execution reasonable?

[Robbepop]

A factor of 2-4x compared to Wasm3 is reasonable but these benchmark results show a factor of 8x compared to Wasm3. So this issue might be valid but further inspection need to happen. Thanks a lot for the report!

Also I think you might have used the WAMR jit engine, since usually the WAMR interpreter performs slightly worse than Wasm3.

[Robbepop]

See my comment here #759 (comment).

[Robbepop]

Closed in favor of #759 since both issues are very much related.

[Robbepop]

@luxinyi0105 I just ran the mutated_file.wat on Wasmi v0.38.0 and it finished execution timely:
(M2 Macbook Pro)

% time wasmi_cli 01-mutated.wat 
wasmi_cli 01-mutated.wat  99.03s user 0.35s system 99% cpu 1:39.56 total