Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

WASM runtime API calls on Windows are very expensive #11275

Closed
2 tasks done
nazar-pc opened this issue Apr 24, 2022 · 11 comments · Fixed by #11232
Closed
2 tasks done

WASM runtime API calls on Windows are very expensive #11275

nazar-pc opened this issue Apr 24, 2022 · 11 comments · Fixed by #11232

Comments

@nazar-pc
Copy link
Contributor

nazar-pc commented Apr 24, 2022

Is there an existing issue?

  • I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

  • This is not a support question.

Description of bug

We discovered in subspace/subspace#307 that sync on Windows is extremely slow. I did some initial testing and discovered that every runtime API call on Windows takes ~5ms, resulting in sync speed of under 20 blocks/s with our chain, while under Linux I can sync with 500 blocks/s speed.

Not sure what the root cause is yet, but it is a UX issue for our chain and it seems to originate from Substrate, hence this issue.

UPD: With native runtime sync on Windows increases to ~350 blocks/s, runtime call cost vanished.

Steps to reproduce

https://github.com/subspace/subspace/blob/main/docs/farming.md can be used to run pre-built Windows binaries of our chain and see sync speed, it also has instructions for building from source in case you want to add instrumentation and recompile. I guess building any other chain for Windows will result in similar sync slowness.

@nazar-pc nazar-pc changed the title Runtime API calls on Windows are very expensive WASM runtime API calls on Windows are very expensive Apr 24, 2022
@burdges
Copy link

burdges commented Apr 24, 2022

You've checked if wasmtime alone exhibits this problem?

@nazar-pc
Copy link
Contributor Author

Just ran wasmtime benches on Linux and Windows (VM on the same machine, but should be close due to hardware acceleration).

Linux:

sync/no-hook/host-to-wasm - typed - nop
                        time:   [16.367 ns 16.614 ns 16.938 ns]
Found 14 outliers among 100 measurements (14.00%)
  6 (6.00%) high mild
  8 (8.00%) high severe
sync/no-hook/host-to-wasm - untyped - nop
                        time:   [27.129 ns 27.570 ns 28.069 ns]
Found 13 outliers among 100 measurements (13.00%)
  6 (6.00%) high mild
  7 (7.00%) high severe
sync/no-hook/host-to-wasm - unchecked - nop
                        time:   [19.478 ns 19.875 ns 20.334 ns]
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) high mild
  7 (7.00%) high severe
sync/no-hook/host-to-wasm - typed - nop-params-and-results
                        time:   [17.184 ns 17.599 ns 18.133 ns]
Found 14 outliers among 100 measurements (14.00%)
  3 (3.00%) high mild
  11 (11.00%) high severe
sync/no-hook/host-to-wasm - untyped - nop-params-and-results
                        time:   [39.586 ns 40.199 ns 40.983 ns]
Found 16 outliers among 100 measurements (16.00%)
  3 (3.00%) high mild
  13 (13.00%) high severe
sync/no-hook/host-to-wasm - unchecked - nop-params-and-results
                        time:   [25.334 ns 25.591 ns 25.913 ns]
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) high mild
  3 (3.00%) high severe

sync/hook-sync/host-to-wasm - typed - nop
                        time:   [16.624 ns 16.763 ns 16.915 ns]
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) high mild
  4 (4.00%) high severe
sync/hook-sync/host-to-wasm - untyped - nop
                        time:   [27.763 ns 28.421 ns 29.224 ns]
Found 17 outliers among 100 measurements (17.00%)
  2 (2.00%) high mild
  15 (15.00%) high severe
sync/hook-sync/host-to-wasm - unchecked - nop
                        time:   [19.417 ns 19.558 ns 19.730 ns]
Found 15 outliers among 100 measurements (15.00%)
  6 (6.00%) high mild
  9 (9.00%) high severe
sync/hook-sync/host-to-wasm - typed - nop-params-and-results
                        time:   [17.293 ns 17.426 ns 17.599 ns]
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) high mild
  7 (7.00%) high severe
sync/hook-sync/host-to-wasm - untyped - nop-params-and-results
                        time:   [41.082 ns 41.996 ns 43.080 ns]
Found 10 outliers among 100 measurements (10.00%)
  1 (1.00%) high mild
  9 (9.00%) high severe
sync/hook-sync/host-to-wasm - unchecked - nop-params-and-results
                        time:   [25.679 ns 26.025 ns 26.580 ns]
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) high mild
  5 (5.00%) high severe

async/no-hook/host-to-wasm - typed - nop
                        time:   [2.7586 us 2.8162 us 2.8828 us]
Found 11 outliers among 100 measurements (11.00%)
  6 (6.00%) high mild
  5 (5.00%) high severe
async/no-hook/host-to-wasm - untyped - nop
                        time:   [2.7718 us 2.7917 us 2.8132 us]
Found 8 outliers among 100 measurements (8.00%)
  7 (7.00%) high mild
  1 (1.00%) high severe
async/no-hook/host-to-wasm - typed - nop-params-and-results
                        time:   [2.8015 us 2.8242 us 2.8492 us]
Found 10 outliers among 100 measurements (10.00%)
  7 (7.00%) high mild
  3 (3.00%) high severe
async/no-hook/host-to-wasm - untyped - nop-params-and-results
                        time:   [2.9011 us 2.9497 us 3.0065 us]
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) high mild
  5 (5.00%) high severe

async/hook-sync/host-to-wasm - typed - nop
                        time:   [2.7605 us 2.7833 us 2.8097 us]
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe
async/hook-sync/host-to-wasm - untyped - nop
                        time:   [2.8342 us 2.8644 us 2.8996 us]
Found 9 outliers among 100 measurements (9.00%)
  3 (3.00%) high mild
  6 (6.00%) high severe
async/hook-sync/host-to-wasm - typed - nop-params-and-results
                        time:   [2.7866 us 2.8347 us 2.8927 us]
Found 13 outliers among 100 measurements (13.00%)
  2 (2.00%) high mild
  11 (11.00%) high severe
async/hook-sync/host-to-wasm - untyped - nop-params-and-results
                        time:   [2.8458 us 2.8836 us 2.9308 us]
Found 7 outliers among 100 measurements (7.00%)
  3 (3.00%) high mild
  4 (4.00%) high severe

async-pool/no-hook/host-to-wasm - typed - nop
                        time:   [1.1402 us 1.1758 us 1.2196 us]
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) high mild
  8 (8.00%) high severe
async-pool/no-hook/host-to-wasm - untyped - nop
                        time:   [1.1675 us 1.1864 us 1.2099 us]
Found 6 outliers among 100 measurements (6.00%)
  6 (6.00%) high severe
async-pool/no-hook/host-to-wasm - typed - nop-params-and-results
                        time:   [1.1331 us 1.1461 us 1.1649 us]
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe
async-pool/no-hook/host-to-wasm - untyped - nop-params-and-results
                        time:   [1.2200 us 1.2331 us 1.2486 us]
Found 6 outliers among 100 measurements (6.00%)
  5 (5.00%) high mild
  1 (1.00%) high severe

async-pool/hook-sync/host-to-wasm - typed - nop
                        time:   [1.1515 us 1.1642 us 1.1793 us]
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) high mild
  4 (4.00%) high severe
async-pool/hook-sync/host-to-wasm - untyped - nop
                        time:   [1.1761 us 1.1908 us 1.2090 us]
Found 14 outliers among 100 measurements (14.00%)
  2 (2.00%) high mild
  12 (12.00%) high severe
async-pool/hook-sync/host-to-wasm - typed - nop-params-and-results
                        time:   [1.1727 us 1.1939 us 1.2181 us]
Found 11 outliers among 100 measurements (11.00%)
  5 (5.00%) high mild
  6 (6.00%) high severe
async-pool/hook-sync/host-to-wasm - untyped - nop-params-and-results
                        time:   [1.2179 us 1.2414 us 1.2715 us]
Found 10 outliers among 100 measurements (10.00%)
  4 (4.00%) high mild
  6 (6.00%) high severe

sync/no-hook/wasm-to-host - nop - typed
                        time:   [4.5954 ns 4.6622 ns 4.7509 ns]
Found 6 outliers among 100 measurements (6.00%)
  4 (4.00%) high mild
  2 (2.00%) high severe
sync/no-hook/wasm-to-host - nop-params-and-results - typed
                        time:   [6.6007 ns 6.6794 ns 6.7765 ns]
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe
sync/no-hook/wasm-to-host - nop - untyped
                        time:   [15.826 ns 16.042 ns 16.294 ns]
Found 16 outliers among 100 measurements (16.00%)
  4 (4.00%) high mild
  12 (12.00%) high severe
sync/no-hook/wasm-to-host - nop-params-and-results - untyped
                        time:   [26.221 ns 26.900 ns 27.663 ns]
Found 16 outliers among 100 measurements (16.00%)
  4 (4.00%) low mild
  7 (7.00%) high mild
  5 (5.00%) high severe
sync/no-hook/wasm-to-host - nop - unchecked
                        time:   [5.9192 ns 6.0208 ns 6.1275 ns]
Found 31 outliers among 100 measurements (31.00%)
  15 (15.00%) low severe
  1 (1.00%) low mild
  6 (6.00%) high mild
  9 (9.00%) high severe
sync/no-hook/wasm-to-host - nop-params-and-results - unchecked
                        time:   [6.8242 ns 6.9810 ns 7.1666 ns]
Found 7 outliers among 100 measurements (7.00%)
  5 (5.00%) high mild
  2 (2.00%) high severe

sync/hook-sync/wasm-to-host - nop - typed
                        time:   [5.0403 ns 5.0872 ns 5.1420 ns]
Found 19 outliers among 100 measurements (19.00%)
  6 (6.00%) high mild
  13 (13.00%) high severe
sync/hook-sync/wasm-to-host - nop-params-and-results - typed
                        time:   [6.9612 ns 7.0587 ns 7.1727 ns]
Found 13 outliers among 100 measurements (13.00%)
  7 (7.00%) high mild
  6 (6.00%) high severe
sync/hook-sync/wasm-to-host - nop - untyped
                        time:   [16.297 ns 16.590 ns 16.916 ns]
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe
sync/hook-sync/wasm-to-host - nop-params-and-results - untyped
                        time:   [27.396 ns 27.933 ns 28.552 ns]
Found 12 outliers among 100 measurements (12.00%)
  5 (5.00%) high mild
  7 (7.00%) high severe
sync/hook-sync/wasm-to-host - nop - unchecked
                        time:   [6.6893 ns 6.7954 ns 6.9083 ns]
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe
sync/hook-sync/wasm-to-host - nop-params-and-results - unchecked
                        time:   [8.0811 ns 8.3195 ns 8.5761 ns]
Found 11 outliers among 100 measurements (11.00%)
  7 (7.00%) high mild
  4 (4.00%) high severe

async/no-hook/wasm-to-host - nop - typed
                        time:   [4.7521 ns 4.8146 ns 4.8897 ns]
Found 8 outliers among 100 measurements (8.00%)
  7 (7.00%) high mild
  1 (1.00%) high severe
async/no-hook/wasm-to-host - nop-params-and-results - typed
                        time:   [6.8724 ns 7.0483 ns 7.2539 ns]
Found 7 outliers among 100 measurements (7.00%)
  6 (6.00%) high mild
  1 (1.00%) high severe
async/no-hook/wasm-to-host - nop - untyped
                        time:   [16.378 ns 16.843 ns 17.369 ns]
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe
async/no-hook/wasm-to-host - nop-params-and-results - untyped
                        time:   [28.281 ns 28.990 ns 29.806 ns]
async/no-hook/wasm-to-host - nop - unchecked
                        time:   [5.7917 ns 5.8958 ns 6.0115 ns]
Found 13 outliers among 100 measurements (13.00%)
  10 (10.00%) low mild
  1 (1.00%) high mild
  2 (2.00%) high severe
async/no-hook/wasm-to-host - nop-params-and-results - unchecked
                        time:   [7.3351 ns 7.5264 ns 7.7254 ns]
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high mild
async/no-hook/wasm-to-host - nop - async-typed
                        time:   [19.866 ns 20.089 ns 20.323 ns]
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe
async/no-hook/wasm-to-host - nop-params-and-results - async-typed
                        time:   [20.418 ns 20.839 ns 21.309 ns]
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild

async/hook-sync/wasm-to-host - nop - typed
                        time:   [5.2855 ns 5.3649 ns 5.4514 ns]
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) high mild
  6 (6.00%) high severe
async/hook-sync/wasm-to-host - nop-params-and-results - typed
                        time:   [7.2675 ns 7.4241 ns 7.6046 ns]
Found 6 outliers among 100 measurements (6.00%)
  5 (5.00%) high mild
  1 (1.00%) high severe
async/hook-sync/wasm-to-host - nop - untyped
                        time:   [16.663 ns 16.986 ns 17.364 ns]
Found 9 outliers among 100 measurements (9.00%)
  3 (3.00%) high mild
  6 (6.00%) high severe
async/hook-sync/wasm-to-host - nop-params-and-results - untyped
                        time:   [29.214 ns 30.043 ns 30.917 ns]
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild
async/hook-sync/wasm-to-host - nop - unchecked
                        time:   [6.3967 ns 6.5125 ns 6.6315 ns]
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild
async/hook-sync/wasm-to-host - nop-params-and-results - unchecked
                        time:   [7.0494 ns 7.2080 ns 7.4040 ns]
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe
async/hook-sync/wasm-to-host - nop - async-typed
                        time:   [19.690 ns 19.913 ns 20.177 ns]
Found 6 outliers among 100 measurements (6.00%)
  6 (6.00%) high mild
async/hook-sync/wasm-to-host - nop-params-and-results - async-typed
                        time:   [20.318 ns 20.684 ns 21.093 ns]
Found 11 outliers among 100 measurements (11.00%)
  4 (4.00%) high mild
  7 (7.00%) high severe

async-pool/no-hook/wasm-to-host - nop - typed
                        time:   [4.6467 ns 4.7436 ns 4.8617 ns]
Found 12 outliers among 100 measurements (12.00%)
  5 (5.00%) high mild
  7 (7.00%) high severe
async-pool/no-hook/wasm-to-host - nop-params-and-results - typed
                        time:   [6.3266 ns 6.3897 ns 6.4677 ns]
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe
async-pool/no-hook/wasm-to-host - nop - untyped
                        time:   [15.148 ns 15.447 ns 15.770 ns]
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe
async-pool/no-hook/wasm-to-host - nop-params-and-results - untyped
                        time:   [26.297 ns 26.762 ns 27.285 ns]
Found 12 outliers among 100 measurements (12.00%)
  10 (10.00%) high mild
  2 (2.00%) high severe
async-pool/no-hook/wasm-to-host - nop - unchecked
                        time:   [5.9248 ns 5.9639 ns 6.0041 ns]
Found 15 outliers among 100 measurements (15.00%)
  4 (4.00%) low severe
  2 (2.00%) high mild
  9 (9.00%) high severe
async-pool/no-hook/wasm-to-host - nop-params-and-results - unchecked
                        time:   [6.8554 ns 6.9806 ns 7.1197 ns]
Found 6 outliers among 100 measurements (6.00%)
  5 (5.00%) high mild
  1 (1.00%) high severe
async-pool/no-hook/wasm-to-host - nop - async-typed
                        time:   [19.139 ns 19.450 ns 19.822 ns]
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe
async-pool/no-hook/wasm-to-host - nop-params-and-results - async-typed
                        time:   [18.990 ns 19.198 ns 19.464 ns]
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe

async-pool/hook-sync/wasm-to-host - nop - typed
                        time:   [5.1559 ns 5.2294 ns 5.3208 ns]
Found 8 outliers among 100 measurements (8.00%)
  8 (8.00%) high mild
async-pool/hook-sync/wasm-to-host - nop-params-and-results - typed
                        time:   [7.2271 ns 7.3529 ns 7.4908 ns]
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) high mild
  4 (4.00%) high severe
async-pool/hook-sync/wasm-to-host - nop - untyped
                        time:   [16.045 ns 16.467 ns 16.925 ns]
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe
async-pool/hook-sync/wasm-to-host - nop-params-and-results - untyped
                        time:   [27.811 ns 28.317 ns 28.883 ns]
Found 12 outliers among 100 measurements (12.00%)
  7 (7.00%) high mild
  5 (5.00%) high severe
async-pool/hook-sync/wasm-to-host - nop - unchecked
                        time:   [6.5011 ns 6.5984 ns 6.7094 ns]
Found 21 outliers among 100 measurements (21.00%)
  12 (12.00%) low mild
  3 (3.00%) high mild
  6 (6.00%) high severe
async-pool/hook-sync/wasm-to-host - nop-params-and-results - unchecked
                        time:   [7.8650 ns 8.0823 ns 8.3586 ns]
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe
async-pool/hook-sync/wasm-to-host - nop - async-typed
                        time:   [19.444 ns 20.015 ns 20.960 ns]
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe
async-pool/hook-sync/wasm-to-host - nop-params-and-results - async-typed
                        time:   [19.980 ns 20.422 ns 20.960 ns]
Found 14 outliers among 100 measurements (14.00%)
  7 (7.00%) high mild
  7 (7.00%) high severe

Windows:

sync/no-hook/host-to-wasm - typed - nop
                        time:   [20.931 ns 21.277 ns 21.659 ns]
Found 8 outliers among 100 measurements (8.00%)
  4 (4.00%) high mild
  4 (4.00%) high severe
sync/no-hook/host-to-wasm - untyped - nop
                        time:   [33.197 ns 33.952 ns 34.806 ns]
Found 14 outliers among 100 measurements (14.00%)
  7 (7.00%) high mild
  7 (7.00%) high severe
sync/no-hook/host-to-wasm - unchecked - nop
                        time:   [25.033 ns 25.289 ns 25.587 ns]
Found 15 outliers among 100 measurements (15.00%)
  7 (7.00%) high mild
  8 (8.00%) high severe
Benchmarking sync/no-hook/host-to-wasm - typed - nop-params-and-results: Collecting 100 samples in estimated 5.0001 s (2                                                                                                                        sync/no-hook/host-to-wasm - typed - nop-params-and-results
                        time:   [25.033 ns 25.647 ns 26.349 ns]
Found 12 outliers among 100 measurements (12.00%)
  4 (4.00%) high mild
  8 (8.00%) high severe
Benchmarking sync/no-hook/host-to-wasm - untyped - nop-params-and-results: Collecting 100 samples in estimated 5.0001 s                                                                                                                         sync/no-hook/host-to-wasm - untyped - nop-params-and-results
                        time:   [45.481 ns 45.985 ns 46.580 ns]
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) high mild
  5 (5.00%) high severe
Benchmarking sync/no-hook/host-to-wasm - unchecked - nop-params-and-results: Collecting 100 samples in estimated 5.0000                                                                                                                         sync/no-hook/host-to-wasm - unchecked - nop-params-and-results
                        time:   [28.588 ns 29.141 ns 29.798 ns]
Found 12 outliers among 100 measurements (12.00%)
  3 (3.00%) high mild
  9 (9.00%) high severe

sync/hook-sync/host-to-wasm - typed - nop
                        time:   [21.634 ns 21.950 ns 22.328 ns]
Found 8 outliers among 100 measurements (8.00%)
  4 (4.00%) high mild
  4 (4.00%) high severe
sync/hook-sync/host-to-wasm - untyped - nop
                        time:   [33.333 ns 33.993 ns 34.716 ns]
Found 13 outliers among 100 measurements (13.00%)
  6 (6.00%) high mild
  7 (7.00%) high severe
Benchmarking sync/hook-sync/host-to-wasm - unchecked - nop: Collecting 100 samples in estimated 5.0001 s (195M iteration                                                                                                                        sync/hook-sync/host-to-wasm - unchecked - nop
                        time:   [25.530 ns 26.061 ns 26.678 ns]
Found 14 outliers among 100 measurements (14.00%)
  7 (7.00%) high mild
  7 (7.00%) high severe
Benchmarking sync/hook-sync/host-to-wasm - typed - nop-params-and-results: Collecting 100 samples in estimated 5.0000 s                                                                                                                         sync/hook-sync/host-to-wasm - typed - nop-params-and-results
                        time:   [24.275 ns 24.584 ns 24.940 ns]
Found 8 outliers among 100 measurements (8.00%)
  4 (4.00%) high mild
  4 (4.00%) high severe
Benchmarking sync/hook-sync/host-to-wasm - untyped - nop-params-and-results: Collecting 100 samples in estimated 5.0002                                                                                                                         sync/hook-sync/host-to-wasm - untyped - nop-params-and-results
                        time:   [45.354 ns 46.244 ns 47.330 ns]
Found 11 outliers among 100 measurements (11.00%)
  7 (7.00%) high mild
  4 (4.00%) high severe
Benchmarking sync/hook-sync/host-to-wasm - unchecked - nop-params-and-results: Collecting 100 samples in estimated 5.000                                                                                                                        sync/hook-sync/host-to-wasm - unchecked - nop-params-and-results
                        time:   [29.469 ns 29.771 ns 30.117 ns]
Found 12 outliers among 100 measurements (12.00%)
  4 (4.00%) high mild
  8 (8.00%) high severe

async/no-hook/host-to-wasm - typed - nop
                        time:   [9.5846 us 9.8007 us 10.065 us]
Found 14 outliers among 100 measurements (14.00%)
  7 (7.00%) high mild
  7 (7.00%) high severe
async/no-hook/host-to-wasm - untyped - nop
                        time:   [9.5207 us 9.6544 us 9.7991 us]
Found 10 outliers among 100 measurements (10.00%)
  5 (5.00%) high mild
  5 (5.00%) high severe
Benchmarking async/no-hook/host-to-wasm - typed - nop-params-and-results: Collecting 100 samples in estimated 5.0485 s (                                                                                                                        async/no-hook/host-to-wasm - typed - nop-params-and-results
                        time:   [9.5048 us 9.6225 us 9.7703 us]
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) high mild
  5 (5.00%) high severe
Benchmarking async/no-hook/host-to-wasm - untyped - nop-params-and-results: Collecting 100 samples in estimated 5.0193 s                                                                                                                        async/no-hook/host-to-wasm - untyped - nop-params-and-results
                        time:   [9.4841 us 9.6254 us 9.8235 us]
Found 11 outliers among 100 measurements (11.00%)
  4 (4.00%) high mild
  7 (7.00%) high severe

async/hook-sync/host-to-wasm - typed - nop
                        time:   [9.2917 us 9.4176 us 9.5857 us]
Found 7 outliers among 100 measurements (7.00%)
  6 (6.00%) high mild
  1 (1.00%) high severe
Benchmarking async/hook-sync/host-to-wasm - untyped - nop: Collecting 100 samples in estimated 5.0455 s (535k iterations                                                                                                                        async/hook-sync/host-to-wasm - untyped - nop
                        time:   [9.3739 us 9.5118 us 9.6753 us]
Found 13 outliers among 100 measurements (13.00%)
  3 (3.00%) high mild
  10 (10.00%) high severe
Benchmarking async/hook-sync/host-to-wasm - typed - nop-params-and-results: Collecting 100 samples in estimated 5.0305 s                                                                                                                        async/hook-sync/host-to-wasm - typed - nop-params-and-results
                        time:   [9.5389 us 9.6939 us 9.8892 us]
Found 10 outliers among 100 measurements (10.00%)
  5 (5.00%) high mild
  5 (5.00%) high severe
Benchmarking async/hook-sync/host-to-wasm - untyped - nop-params-and-results: Collecting 100 samples in estimated 5.0248                                                                                                                        async/hook-sync/host-to-wasm - untyped - nop-params-and-results
                        time:   [9.3830 us 9.5365 us 9.7369 us]
Found 15 outliers among 100 measurements (15.00%)
  7 (7.00%) high mild
  8 (8.00%) high severe

Benchmarking async-pool/no-hook/host-to-wasm - typed - nop: Collecting 100 samples in estimated 5.0348 s (535k iteration                                                                                                                        async-pool/no-hook/host-to-wasm - typed - nop
                        time:   [9.3826 us 9.5810 us 9.8138 us]
Found 10 outliers among 100 measurements (10.00%)
  3 (3.00%) high mild
  7 (7.00%) high severe
Benchmarking async-pool/no-hook/host-to-wasm - untyped - nop: Collecting 100 samples in estimated 5.0161 s (520k iterati                                                                                                                        async-pool/no-hook/host-to-wasm - untyped - nop
                        time:   [9.5500 us 9.6966 us 9.8549 us]
Found 13 outliers among 100 measurements (13.00%)
  5 (5.00%) high mild
  8 (8.00%) high severe
Benchmarking async-pool/no-hook/host-to-wasm - typed - nop-params-and-results: Collecting 100 samples in estimated 5.046                                                                                                                        async-pool/no-hook/host-to-wasm - typed - nop-params-and-results
                        time:   [9.3686 us 9.4760 us 9.6048 us]
Found 10 outliers among 100 measurements (10.00%)
  6 (6.00%) high mild
  4 (4.00%) high severe
Benchmarking async-pool/no-hook/host-to-wasm - untyped - nop-params-and-results: Collecting 100 samples in estimated 5.0                                                                                                                        async-pool/no-hook/host-to-wasm - untyped - nop-params-and-results
                        time:   [9.4108 us 9.5347 us 9.6828 us]
Found 10 outliers among 100 measurements (10.00%)
  2 (2.00%) high mild
  8 (8.00%) high severe

Benchmarking async-pool/hook-sync/host-to-wasm - typed - nop: Collecting 100 samples in estimated 5.0265 s (520k iterati                                                                                                                        async-pool/hook-sync/host-to-wasm - typed - nop
                        time:   [9.2829 us 9.4234 us 9.6101 us]
Found 12 outliers among 100 measurements (12.00%)
  5 (5.00%) high mild
  7 (7.00%) high severe
Benchmarking async-pool/hook-sync/host-to-wasm - untyped - nop: Collecting 100 samples in estimated 5.0395 s (530k itera                                                                                                                        async-pool/hook-sync/host-to-wasm - untyped - nop
                        time:   [9.4862 us 9.7103 us 9.9551 us]
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) high mild
  6 (6.00%) high severe
Benchmarking async-pool/hook-sync/host-to-wasm - typed - nop-params-and-results: Collecting 100 samples in estimated 5.0                                                                                                                        async-pool/hook-sync/host-to-wasm - typed - nop-params-and-results
                        time:   [9.6818 us 9.8111 us 9.9490 us]
Found 9 outliers among 100 measurements (9.00%)
  3 (3.00%) high mild
  6 (6.00%) high severe
Benchmarking async-pool/hook-sync/host-to-wasm - untyped - nop-params-and-results: Collecting 100 samples in estimated 5                                                                                                                        async-pool/hook-sync/host-to-wasm - untyped - nop-params-and-results
                        time:   [9.4767 us 9.6414 us 9.8352 us]
Found 10 outliers among 100 measurements (10.00%)
  6 (6.00%) high mild
  4 (4.00%) high severe

sync/no-hook/wasm-to-host - nop - typed
                        time:   [4.3459 ns 4.3841 ns 4.4283 ns]
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) high mild
  5 (5.00%) high severe
Benchmarking sync/no-hook/wasm-to-host - nop-params-and-results - typed: Collecting 100 samples in estimated 5.0000 s (6                                                                                                                        sync/no-hook/wasm-to-host - nop-params-and-results - typed
                        time:   [7.8673 ns 7.9658 ns 8.0790 ns]
Found 12 outliers among 100 measurements (12.00%)
  5 (5.00%) high mild
  7 (7.00%) high severe
sync/no-hook/wasm-to-host - nop - untyped
                        time:   [14.785 ns 15.186 ns 15.625 ns]
Found 11 outliers among 100 measurements (11.00%)
  6 (6.00%) high mild
  5 (5.00%) high severe
Benchmarking sync/no-hook/wasm-to-host - nop-params-and-results - untyped: Collecting 100 samples in estimated 5.0001 s                                                                                                                         sync/no-hook/wasm-to-host - nop-params-and-results - untyped
                        time:   [30.894 ns 31.172 ns 31.498 ns]
Found 15 outliers among 100 measurements (15.00%)
  7 (7.00%) high mild
  8 (8.00%) high severe
sync/no-hook/wasm-to-host - nop - unchecked
                        time:   [6.8107 ns 6.9696 ns 7.1575 ns]
Found 7 outliers among 100 measurements (7.00%)
  5 (5.00%) high mild
  2 (2.00%) high severe
Benchmarking sync/no-hook/wasm-to-host - nop-params-and-results - unchecked: Collecting 100 samples in estimated 5.0000                                                                                                                         sync/no-hook/wasm-to-host - nop-params-and-results - unchecked
                        time:   [7.4883 ns 7.6253 ns 7.7836 ns]
Found 26 outliers among 100 measurements (26.00%)
  1 (1.00%) low severe
  5 (5.00%) low mild
  3 (3.00%) high mild
  17 (17.00%) high severe

sync/hook-sync/wasm-to-host - nop - typed
                        time:   [4.8029 ns 4.9226 ns 5.0741 ns]
Found 13 outliers among 100 measurements (13.00%)
  5 (5.00%) high mild
  8 (8.00%) high severe
Benchmarking sync/hook-sync/wasm-to-host - nop-params-and-results - typed: Collecting 100 samples in estimated 5.0000 s                                                                                                                         sync/hook-sync/wasm-to-host - nop-params-and-results - typed
                        time:   [7.8820 ns 7.9902 ns 8.1212 ns]
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) high mild
  3 (3.00%) high severe
sync/hook-sync/wasm-to-host - nop - untyped
                        time:   [15.941 ns 16.242 ns 16.621 ns]
Found 17 outliers among 100 measurements (17.00%)
  7 (7.00%) high mild
  10 (10.00%) high severe
Benchmarking sync/hook-sync/wasm-to-host - nop-params-and-results - untyped: Collecting 100 samples in estimated 5.0001                                                                                                                         sync/hook-sync/wasm-to-host - nop-params-and-results - untyped
                        time:   [31.853 ns 32.304 ns 32.859 ns]
Found 14 outliers among 100 measurements (14.00%)
  4 (4.00%) high mild
  10 (10.00%) high severe
Benchmarking sync/hook-sync/wasm-to-host - nop - unchecked: Collecting 100 samples in estimated 5.0000 s (683M iteration                                                                                                                        sync/hook-sync/wasm-to-host - nop - unchecked
                        time:   [7.4150 ns 7.5144 ns 7.6200 ns]
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe
Benchmarking sync/hook-sync/wasm-to-host - nop-params-and-results - unchecked: Collecting 100 samples in estimated 5.000                                                                                                                        sync/hook-sync/wasm-to-host - nop-params-and-results - unchecked
                        time:   [7.9295 ns 8.0062 ns 8.0912 ns]
Found 17 outliers among 100 measurements (17.00%)
  7 (7.00%) high mild
  10 (10.00%) high severe

async/no-hook/wasm-to-host - nop - typed
                        time:   [4.3477 ns 4.3874 ns 4.4360 ns]
Found 14 outliers among 100 measurements (14.00%)
  6 (6.00%) high mild
  8 (8.00%) high severe
Benchmarking async/no-hook/wasm-to-host - nop-params-and-results - typed: Collecting 100 samples in estimated 5.0000 s (                                                                                                                        async/no-hook/wasm-to-host - nop-params-and-results - typed
                        time:   [7.2414 ns 7.3592 ns 7.4874 ns]
Found 8 outliers among 100 measurements (8.00%)
  7 (7.00%) high mild
  1 (1.00%) high severe
async/no-hook/wasm-to-host - nop - untyped
                        time:   [14.474 ns 14.707 ns 14.976 ns]
Found 10 outliers among 100 measurements (10.00%)
  5 (5.00%) high mild
  5 (5.00%) high severe
Benchmarking async/no-hook/wasm-to-host - nop-params-and-results - untyped: Collecting 100 samples in estimated 5.0001 s                                                                                                                        async/no-hook/wasm-to-host - nop-params-and-results - untyped
                        time:   [31.463 ns 31.895 ns 32.395 ns]
Found 7 outliers among 100 measurements (7.00%)
  5 (5.00%) high mild
  2 (2.00%) high severe
Benchmarking async/no-hook/wasm-to-host - nop - unchecked: Collecting 100 samples in estimated 5.0000 s (764M iterations                                                                                                                        async/no-hook/wasm-to-host - nop - unchecked
                        time:   [6.6385 ns 6.7313 ns 6.8335 ns]
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild
Benchmarking async/no-hook/wasm-to-host - nop-params-and-results - unchecked: Collecting 100 samples in estimated 5.0000                                                                                                                        async/no-hook/wasm-to-host - nop-params-and-results - unchecked
                        time:   [7.3764 ns 7.4480 ns 7.5474 ns]
Found 13 outliers among 100 measurements (13.00%)
  3 (3.00%) high mild
  10 (10.00%) high severe
Benchmarking async/no-hook/wasm-to-host - nop - async-typed: Collecting 100 samples in estimated 5.0000 s (120M iteratio                                                                                                                        async/no-hook/wasm-to-host - nop - async-typed
                        time:   [40.872 ns 41.687 ns 42.818 ns]
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe
Benchmarking async/no-hook/wasm-to-host - nop-params-and-results - async-typed: Collecting 100 samples in estimated 5.00                                                                                                                        async/no-hook/wasm-to-host - nop-params-and-results - async-typed
                        time:   [45.041 ns 46.106 ns 47.370 ns]
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high severe

async/hook-sync/wasm-to-host - nop - typed
                        time:   [4.7895 ns 4.8513 ns 4.9282 ns]
Found 22 outliers among 100 measurements (22.00%)
  2 (2.00%) low mild
  2 (2.00%) high mild
  18 (18.00%) high severe
Benchmarking async/hook-sync/wasm-to-host - nop-params-and-results - typed: Collecting 100 samples in estimated 5.0000 s                                                                                                                        async/hook-sync/wasm-to-host - nop-params-and-results - typed
                        time:   [7.8623 ns 7.9662 ns 8.0953 ns]
Found 17 outliers among 100 measurements (17.00%)
  9 (9.00%) high mild
  8 (8.00%) high severe
Benchmarking async/hook-sync/wasm-to-host - nop - untyped: Collecting 100 samples in estimated 5.0000 s (305M iterations                                                                                                                        async/hook-sync/wasm-to-host - nop - untyped
                        time:   [15.882 ns 16.219 ns 16.659 ns]
Found 12 outliers among 100 measurements (12.00%)
  3 (3.00%) high mild
  9 (9.00%) high severe
Benchmarking async/hook-sync/wasm-to-host - nop-params-and-results - untyped: Collecting 100 samples in estimated 5.0001                                                                                                                        async/hook-sync/wasm-to-host - nop-params-and-results - untyped
                        time:   [31.509 ns 31.998 ns 32.605 ns]
Found 9 outliers among 100 measurements (9.00%)
  3 (3.00%) high mild
  6 (6.00%) high severe
Benchmarking async/hook-sync/wasm-to-host - nop - unchecked: Collecting 100 samples in estimated 5.0000 s (691M iteratio                                                                                                                        async/hook-sync/wasm-to-host - nop - unchecked
                        time:   [7.2724 ns 7.3786 ns 7.5089 ns]
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) high mild
  3 (3.00%) high severe
Benchmarking async/hook-sync/wasm-to-host - nop-params-and-results - unchecked: Collecting 100 samples in estimated 5.00                                                                                                                        async/hook-sync/wasm-to-host - nop-params-and-results - unchecked
                        time:   [7.7471 ns 7.8431 ns 7.9627 ns]
Found 18 outliers among 100 measurements (18.00%)
  6 (6.00%) high mild
  12 (12.00%) high severe
Benchmarking async/hook-sync/wasm-to-host - nop - async-typed: Collecting 100 samples in estimated 5.0002 s (115M iterat                                                                                                                        async/hook-sync/wasm-to-host - nop - async-typed
                        time:   [44.058 ns 44.734 ns 45.519 ns]
Found 12 outliers among 100 measurements (12.00%)
  6 (6.00%) high mild
  6 (6.00%) high severe
Benchmarking async/hook-sync/wasm-to-host - nop-params-and-results - async-typed: Collecting 100 samples in estimated 5.                                                                                                                        async/hook-sync/wasm-to-host - nop-params-and-results - async-typed
                        time:   [45.229 ns 46.539 ns 48.218 ns]
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe

Benchmarking async-pool/no-hook/wasm-to-host - nop - typed: Collecting 100 samples in estimated 5.0000 s (1.1B iteration                                                                                                                        async-pool/no-hook/wasm-to-host - nop - typed
                        time:   [4.3902 ns 4.4443 ns 4.5162 ns]
Found 15 outliers among 100 measurements (15.00%)
  7 (7.00%) high mild
  8 (8.00%) high severe
Benchmarking async-pool/no-hook/wasm-to-host - nop-params-and-results - typed: Collecting 100 samples in estimated 5.000                                                                                                                        async-pool/no-hook/wasm-to-host - nop-params-and-results - typed
                        time:   [7.1903 ns 7.2806 ns 7.3864 ns]
Found 11 outliers among 100 measurements (11.00%)
  6 (6.00%) high mild
  5 (5.00%) high severe
Benchmarking async-pool/no-hook/wasm-to-host - nop - untyped: Collecting 100 samples in estimated 5.0001 s (348M iterati                                                                                                                        async-pool/no-hook/wasm-to-host - nop - untyped
                        time:   [14.453 ns 14.616 ns 14.811 ns]
Found 11 outliers among 100 measurements (11.00%)
  6 (6.00%) high mild
  5 (5.00%) high severe
Benchmarking async-pool/no-hook/wasm-to-host - nop-params-and-results - untyped: Collecting 100 samples in estimated 5.0                                                                                                                        async-pool/no-hook/wasm-to-host - nop-params-and-results - untyped
                        time:   [31.135 ns 31.900 ns 32.837 ns]
Found 15 outliers among 100 measurements (15.00%)
  7 (7.00%) high mild
  8 (8.00%) high severe
Benchmarking async-pool/no-hook/wasm-to-host - nop - unchecked: Collecting 100 samples in estimated 5.0000 s (774M itera                                                                                                                        async-pool/no-hook/wasm-to-host - nop - unchecked
                        time:   [6.5510 ns 6.6342 ns 6.7285 ns]
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) high mild
  6 (6.00%) high severe
Benchmarking async-pool/no-hook/wasm-to-host - nop-params-and-results - unchecked: Collecting 100 samples in estimated 5                                                                                                                        async-pool/no-hook/wasm-to-host - nop-params-and-results - unchecked
                        time:   [7.4157 ns 7.4984 ns 7.5954 ns]
Found 14 outliers among 100 measurements (14.00%)
  7 (7.00%) high mild
  7 (7.00%) high severe
Benchmarking async-pool/no-hook/wasm-to-host - nop - async-typed: Collecting 100 samples in estimated 5.0000 s (119M ite                                                                                                                        async-pool/no-hook/wasm-to-host - nop - async-typed
                        time:   [40.347 ns 40.874 ns 41.480 ns]
Found 11 outliers among 100 measurements (11.00%)
  5 (5.00%) high mild
  6 (6.00%) high severe
Benchmarking async-pool/no-hook/wasm-to-host - nop-params-and-results - async-typed: Collecting 100 samples in estimated                                                                                                                        async-pool/no-hook/wasm-to-host - nop-params-and-results - async-typed
                        time:   [42.651 ns 43.175 ns 43.780 ns]
Found 9 outliers among 100 measurements (9.00%)
  4 (4.00%) high mild
  5 (5.00%) high severe

Benchmarking async-pool/hook-sync/wasm-to-host - nop - typed: Collecting 100 samples in estimated 5.0000 s (1.0B iterati                                                                                                                        async-pool/hook-sync/wasm-to-host - nop - typed
                        time:   [4.7712 ns 4.8183 ns 4.8703 ns]
Found 18 outliers among 100 measurements (18.00%)
  9 (9.00%) high mild
  9 (9.00%) high severe
Benchmarking async-pool/hook-sync/wasm-to-host - nop-params-and-results - typed: Collecting 100 samples in estimated 5.0                                                                                                                        async-pool/hook-sync/wasm-to-host - nop-params-and-results - typed
                        time:   [7.9790 ns 8.0897 ns 8.2233 ns]
Found 12 outliers among 100 measurements (12.00%)
  4 (4.00%) high mild
  8 (8.00%) high severe
Benchmarking async-pool/hook-sync/wasm-to-host - nop - untyped: Collecting 100 samples in estimated 5.0001 s (305M itera                                                                                                                        async-pool/hook-sync/wasm-to-host - nop - untyped
                        time:   [16.496 ns 16.891 ns 17.348 ns]
Found 13 outliers among 100 measurements (13.00%)
  9 (9.00%) high mild
  4 (4.00%) high severe
Benchmarking async-pool/hook-sync/wasm-to-host - nop-params-and-results - untyped: Collecting 100 samples in estimated 5                                                                                                                        async-pool/hook-sync/wasm-to-host - nop-params-and-results - untyped
                        time:   [32.401 ns 33.026 ns 33.746 ns]
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe
Benchmarking async-pool/hook-sync/wasm-to-host - nop - unchecked: Collecting 100 samples in estimated 5.0000 s (685M ite                                                                                                                        async-pool/hook-sync/wasm-to-host - nop - unchecked
                        time:   [7.2158 ns 7.3274 ns 7.4594 ns]
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe
Benchmarking async-pool/hook-sync/wasm-to-host - nop-params-and-results - unchecked: Collecting 100 samples in estimated                                                                                                                        async-pool/hook-sync/wasm-to-host - nop-params-and-results - unchecked
                        time:   [7.8923 ns 8.0183 ns 8.1676 ns]
Found 15 outliers among 100 measurements (15.00%)
  6 (6.00%) high mild
  9 (9.00%) high severe
Benchmarking async-pool/hook-sync/wasm-to-host - nop - async-typed: Collecting 100 samples in estimated 5.0001 s (118M i                                                                                                                        async-pool/hook-sync/wasm-to-host - nop - async-typed
                        time:   [41.091 ns 41.774 ns 42.570 ns]
Found 11 outliers among 100 measurements (11.00%)
  5 (5.00%) high mild
  6 (6.00%) high severe
Benchmarking async-pool/hook-sync/wasm-to-host - nop-params-and-results - async-typed: Collecting 100 samples in estimat                                                                                                                        async-pool/hook-sync/wasm-to-host - nop-params-and-results - async-typed
                        time:   [43.882 ns 44.587 ns 45.443 ns]
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe

I can see Windows almost always slower, with host-to-wasm and wasm-to-host benches being the most affected, around 8x slower in some cases.

However this doesn't explain the difference between 20 and 350 blocks/s fully where the difference is much more severe than in benches.

@bkchr
Copy link
Member

bkchr commented Apr 25, 2022

Do you compile it natively for Windows or do you use the Linux Subsystem under Windows?

@nazar-pc
Copy link
Contributor Author

Natively, x86_64-pc-windows-msvc

@bkchr
Copy link
Member

bkchr commented Apr 25, 2022

And with the Linux Subsystem?

Point here being we don't really support Windows. So, we will not prevent you to fix it and we can merge it, but there will not be that much help from our side.

@nazar-pc
Copy link
Contributor Author

For our use case (we're wrapping Substrate-based node into GUI desktop application among other things) WSL is not an option, so I'll not bother trying that.

Point here being we don't really support Windows. So, we will not prevent you to fix it and we can merge it, but there will not be that much help from our side.

Fair, we'll have to fix that for sure, I'd be thankful for any pointers or things to look into first if possible.

@bkchr
Copy link
Member

bkchr commented Apr 25, 2022

I think @koute can you help here. I think he has some benchmarks for the wasm initialization etc.

@koute
Copy link
Contributor

koute commented Apr 26, 2022

@nazar-pc This is somewhat to be expected. My guess would be (I haven't actually profiled this on Windows, so it's just a guess) that the slowness you're seeing is a side effect of how we currently clear the memory between instantiations. On Linux we use madvise for this which is accelerated by the kernel (doesn't clear the pages which don't need to be cleared) and asynchronous, while on Windows we essentially use a memset which always clears everything and is synchronous.

Could you try running the code from this PR and see how's the performance? #11232

@nazar-pc
Copy link
Contributor Author

I need to wait a bit for longer sync to kick in and print numbers, but it seems like #11232 makes it A LOT faster on Windows, should be on the same order of magnitude as Linux 🔥

How close is it to landing? I see it is already reviewed.

@nazar-pc
Copy link
Contributor Author

Yeah, #11232 should be updated as such that closes this issue.

@koute
Copy link
Contributor

koute commented Apr 27, 2022

That's good to know!

I didn't expect it to improve the performance on Windows, but I forgot that we were doing a memset there which now doesn't have to be executed, so I guess it does improve performance even on Windows. (:

Yeah, it's almost done. It still needs one more review and then it should be good to go.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants