Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compare benchmarks between rustc stable and nightly channels #507

Closed
wants to merge 3 commits into from

Conversation

Robbepop
Copy link
Collaborator

@Robbepop Robbepop commented Oct 11, 2022

I got suspicious after I saw huge regressions in the ballpark of 30-80% on my local development platform so I just wanted to make sure that it is not my local environment by running those same benchmarks on the CI runners.

edit: Those regressions are very real ...

@paritytech-cicd-pr
Copy link

paritytech-cicd-pr commented Oct 11, 2022

BENCHMARKS

NATIVEWASMTIME
BENCHMARKMASTERPRDIFFMASTERPRDIFFWASMTIME OVERHEAD
execute/
count_until
888.50µs 1.24ms 🔴 40.06% 2.38ms 3.10ms 🔴 30.07% 🔴 149%
execute/
factorial_iterative
400.45ns 542.71ns 🔴 35.46% 993.53ns 1.30µs 🔴 30.67% 🔴 139%
execute/
factorial_recursive
761.07ns 840.76ns 🔴 10.47% 1.56µs 1.71µs 🔴 9.65% 🔴 103%
execute/
fib_iterative
1.94ms 2.55ms 🔴 36.62% 5.12ms 6.72ms 🔴 31.43% 🔴 163%
execute/
fib_recursive
6.88ms 8.46ms 🔴 23.79% 14.30ms 15.56ms 🔴 9.01% 🟡 84%
execute/
global_bump
1.43ms 1.83ms 🔴 27.29% 3.63ms 4.53ms 🔴 24.79% 🔴 148%
execute/
host_calls
31.80µs 32.44µs ⚪ 2.03% 45.05µs 48.33µs 🔴 7.37% 🟢 49%
execute/
memory_fill
1.54ms 2.32ms 🔴 50.88% 4.33ms 5.81ms 🔴 34.03% 🔴 151%
execute/
memory_sum
1.52ms 2.34ms 🔴 53.50% 4.37ms 5.84ms 🔴 33.68% 🔴 150%
execute/
memory_vec_add
3.03ms 4.81ms 🔴 60.16% 8.97ms 12.11ms 🔴 35.19% 🔴 152%
execute/
recursive_is_even
1.30ms 1.35ms 🔴 3.91% 2.60ms 2.69ms 🔴 3.15% 🟡 99%
execute/
recursive_ok
173.10µs 181.42µs 🔴 5.00% 350.49µs 380.08µs 🔴 8.82% 🔴 109%
execute/
recursive_scan
213.44µs 235.58µs 🔴 10.47% 468.96µs 505.36µs 🔴 7.65% 🔴 115%
execute/
recursive_trap
17.29µs 17.99µs 🔴 4.15% 35.50µs 39.14µs 🔴 10.18% 🔴 118%
execute/
regex_redux
665.58µs 871.56µs 🔴 30.94% 1.60ms 2.08ms 🔴 29.51% 🔴 138%
execute/
rev_complement
596.99µs 876.68µs 🔴 46.98% 1.55ms 2.11ms 🔴 35.85% 🔴 140%
execute/
tiny_keccak
473.22µs 833.78µs 🔴 77.02% 1.42ms 1.80ms 🔴 26.52% 🔴 116%
execute/
trunc_f2i
1.07ms 1.31ms 🔴 22.19% 2.73ms 3.49ms 🔴 27.80% 🔴 167%
instantiate/
wasm_kernel
70.04µs 71.45µs ⚪ 2.09% 128.03µs 92.10µs 🟢 -27.74% 🟢 29%
translate/
wasm_kernel
4.88ms 4.72ms 🟢 -3.24% 9.47ms 9.88ms 🔴 3.72% 🔴 109%

Link to pipeline

@Robbepop Robbepop changed the title Compare benchmarks between rustc stable and beta channels Compare benchmarks between rustc stable and nightly channels Oct 12, 2022
@Robbepop Robbepop marked this pull request as draft October 14, 2022 15:27
@Robbepop
Copy link
Collaborator Author

Closing this since the demonstration for rust-lang/rust#102952 has been shown.

@Robbepop Robbepop closed this Oct 23, 2022
@Robbepop Robbepop deleted the rf-benchcmp-stable-beta branch March 11, 2024 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants