Publishing benchmarks for graphs #740

alvicsam · 2023-07-24T14:53:36Z

PR adds gitlab and github jobs that run benchmarks for master and create nice graphs that can be found here

close https://github.com/paritytech/ci_cd/issues/848

paritytech-cicd-pr · 2023-07-24T15:04:08Z

BENCHMARKS

	NATIVE			WASMTIME
BENCHMARK	MASTER	PR	DIFF	MASTER	PR	DIFF	WASMTIME OVERHEAD
`execute/` `bare_call_0`	1.54ms	1.55ms	⚪ -0.36%	1.03ms	1.02ms	⚪ -0.51%	🟢 -34%
`execute/` `bare_call_0/typed`	1.18ms	1.15ms	⚪ -0.78%	646.29µs	641.96µs	⚪ -0.67%	🟢 -44%
`execute/` `bare_call_1`	1.68ms	1.60ms	🟢 -4.05%	1.20ms	1.17ms	🟢 -2.06%	🟢 -27%
`execute/` `bare_call_16`	2.54ms	2.55ms	⚪ -0.21%	3.62ms	3.48ms	🟢 -3.82%	🟢 36%
`execute/` `bare_call_16/typed`	1.62ms	1.61ms	⚪ -0.59%	1.81ms	1.76ms	🟢 -2.79%	🟢 10%
`execute/` `bare_call_1/typed`	1.28ms	1.26ms	⚪ -2.18%	940.31µs	943.58µs	⚪ 0.38%	🟢 -25%
`execute/` `bare_call_4`	1.84ms	1.78ms	⚪ -2.73%	1.56ms	1.56ms	⚪ 0.13%	🟢 -12%
`execute/` `bare_call_4/typed`	1.32ms	1.26ms	⚪ -3.73%	921.68µs	921.92µs	⚪ 0.00%	🟢 -27%
`execute/` `br_table`	1.37ms	1.33ms	⚪ -2.02%	1.20ms	1.13ms	🟢 -4.96%	🟢 -15%
`execute/` `count_until`	620.92µs	622.33µs	⚪ 0.12%	1.37ms	1.37ms	⚪ 0.07%	🔴 120%
`execute/` `factorial_iterative`	321.93µs	320.76µs	⚪ -0.30%	687.93µs	686.10µs	⚪ -0.21%	🔴 114%
`execute/` `factorial_recursive`	513.91µs	512.93µs	⚪ -0.20%	852.13µs	892.66µs	🔴 4.59%	🟡 74%
`execute/` `fibonacci_iter`	1.40ms	1.40ms	⚪ 0.13%	3.31ms	3.32ms	⚪ 0.16%	🔴 137%
`execute/` `fibonacci_rec`	4.76ms	4.37ms	🟢 -8.23%	7.87ms	7.78ms	🟢 -1.14%	🟡 78%
`execute/` `fibonacci_tail`	972.98µs	1.04ms	🔴 7.17%	1.89ms	1.89ms	⚪ -0.27%	🟡 81%
`execute/` `global_bump`	716.85µs	724.08µs	⚪ 0.94%	1.67ms	1.67ms	⚪ 0.04%	🔴 130%
`execute/` `global_const`	659.57µs	659.52µs	⚪ -0.03%	1.76ms	1.76ms	⚪ 0.04%	🔴 166%
`execute/` `host_calls`	37.77µs	37.06µs	⚪ -2.57%	38.64µs	39.23µs	🔴 1.48%	🟢 6%
`execute/` `memory_fill`	1.18ms	1.21ms	⚪ 1.30%	2.69ms	2.69ms	⚪ -0.11%	🔴 123%
`execute/` `memory_sum`	1.14ms	1.23ms	🔴 5.21%	2.65ms	2.65ms	⚪ -0.06%	🔴 115%
`execute/` `memory_vec_add`	2.41ms	2.30ms	🟢 -4.05%	5.71ms	5.72ms	⚪ 0.12%	🔴 149%
`execute/` `recursive_is_even`	713.77µs	712.29µs	⚪ 0.22%	1.24ms	1.25ms	⚪ 0.48%	🟡 75%
`execute/` `recursive_ok`	103.18µs	102.26µs	⚪ -0.96%	191.02µs	192.21µs	⚪ 0.65%	🟡 88%
`execute/` `recursive_scan`	138.13µs	137.68µs	⚪ -0.37%	253.96µs	248.39µs	⚪ -1.97%	🟡 80%
`execute/` `recursive_trap`	10.09µs	10.18µs	⚪ 0.86%	19.61µs	19.61µs	⚪ 0.13%	🟡 93%
`execute/` `regex_redux`	455.22µs	459.94µs	⚪ 1.14%	998.79µs	970.46µs	🟢 -2.74%	🔴 111%
`execute/` `rev_complement`	420.27µs	421.40µs	⚪ 0.30%	971.99µs	965.04µs	⚪ -0.75%	🔴 129%
`execute/` `tiny_keccak`	317.14µs	315.10µs	⚪ -0.60%	859.12µs	862.28µs	⚪ 0.27%	🔴 174%
`execute/` `trunc_f2i`	737.53µs	739.33µs	⚪ 0.23%	1.86ms	1.85ms	⚪ -0.26%	🔴 151%
`instantiate/` `wasm_kernel`	54.18µs	53.08µs	🟢 -3.47%	56.32µs	59.74µs	🔴 6.60%	🟢 13%
`translate/` `erc1155`	188.48µs	186.89µs	⚪ 0.05%	306.69µs	311.50µs	🔴 1.72%	🟡 67%
`translate/` `erc20`	91.20µs	91.70µs	⚪ 0.41%	151.07µs	152.58µs	⚪ 0.93%	🟡 66%
`translate/` `erc721`	128.98µs	128.75µs	⚪ -0.32%	215.97µs	218.26µs	⚪ 0.58%	🟡 70%
`translate/` `spidermonkey`	56.13ms	56.16ms	⚪ 0.05%	0.00ns	0.00ns	⚪ 0.69%	🟢 -100%
`translate/` `wasm_kernel`	3.76ms	3.75ms	⚪ -0.01%	5.73ms	5.76ms	⚪ 0.67%	🟡 54%

Link to pipeline

codecov-commenter · 2023-07-25T12:57:45Z

Codecov Report

Merging #740 (91b7386) into master (4860ecc) will increase coverage by 0.15%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #740      +/-   ##
==========================================
+ Coverage   79.25%   79.40%   +0.15%     
==========================================
  Files         104      104              
  Lines        9046     9054       +8     
==========================================
+ Hits         7169     7189      +20     
+ Misses       1877     1865      -12

see 16 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Robbepop · 2023-07-26T16:21:11Z

@alvicsam thanks for implementing this so quickly and sorry for replying to late!

The graphs look pretty cool and the website display is clean.

I have a few extra requests if possible. 😇

Can I haz dark mode eventually (if not much work)? 😅
In most/all graphs the ns/iter column hosts a ton of zeros. Maybe if we use microseconds per iteration (us/iter) it would look a bit less overwhelming.
Maybe it a local problem with my screen but only at 50% zoom level I have a decent level of zoom on the graphs. At 100% I am basically looking at a single graph per page and I think this is simply too big given that we have so many of those graphs. We definitely want a zoom, but default zoom should provide a nice overview as in this picture:

With the zoom talk above it would be really cool with smaller zoom to have a high-level overview of all graphs and make it possible to click on them to enlarge a single graph for inspection if necessary. Is that possible?

alvicsam · 2023-07-27T14:06:39Z

Can I haz dark mode eventually (if not much work)? 😅

Unfortunately I couldn't find this option in the GHA

In most/all graphs the ns/iter column hosts a ton of zeros. Maybe if we use microseconds per iteration (us/iter) it would look a bit less overwhelming.

With this action it's not possible, it's hardcoded there

Maybe it a local problem with my screen but only at 50% zoom level I have a decent level of zoom on the graphs.

No, this is how GHA generates the page with result. I've checked and have the same output: 2 columns with graphs on a big screen and 1 column on my laptop

Robbepop · 2023-07-29T08:56:21Z

Can I haz dark mode eventually (if not much work)? 😅

Unfortunately I couldn't find this option in the GHA

Would it maybe be possible to write feature request issues for this? Not that important though.

In most/all graphs the ns/iter column hosts a ton of zeros. Maybe if we use microseconds per iteration (us/iter) it would look a bit less overwhelming.

With this action it's not possible, it's hardcoded there

Hmm, again I think we could write feature request issues since the ns/iter is really not a great default for everything anyways imo. Sometimes for example you measure throughput or measure network stuff which usually goes with ms/iter.

Robbepop

Thanks a lot for implementing this so quickly @alvicsam! Sorry for taking unusually long to reply and review.

Robbepop · 2023-07-29T08:57:57Z

.gitlab-ci.yml

-.docker-env:                       &docker-env
-  image:                           "paritytech/ci-linux:staging"
+.docker-env:
+  image:                           "paritytech/ci-unified:bullseye-1.71.0"


What are the effects of this and why was this changed/needed?

We replaced ci-linux with ci-unified. It has transparent tags naming, flexible choice of
Rust versions, and so on. More - https://hub.docker.com/repository/docker/paritytech/ci-unified

We are switching to our new ci-unified image. We decided to have "one for all" image and finally added versioning to it.

Robbepop · 2023-07-31T08:31:52Z

@alvicsam Is this ready for merge?

alvicsam · 2023-07-31T10:05:05Z

@alvicsam Is this ready for merge?

yes

Robbepop · 2023-07-31T10:29:02Z

Thanks a lot @alvicsam for this! 🚀
Looking forward to seeing the benchmark graphs over time.

[Do not merge] Publishing benchmarks for graphs

0ac0ceb

alvicsam added 15 commits July 24, 2023 17:21

add wasmtime-benchmark-master

a2cce1a

add dbg ref

0a32242

add collect artifacts

4374c44

switch ci image

6ca6b41

add publish

9472c33

cp instaed mv

70bc57b

add gha

88e8dce

fix on

e58b12a

add gh-pages

f88437c

disable ref for checkout

5f8fa38

trim whitespaces

10d1046

add cancel for previous runs

a4002aa

add gh token

7d36e94

downgrade checkout action

2c8e1d1

add skip-fetch-gh-pages option

c182202

alvicsam added 10 commits July 25, 2023 15:06

debug gha

9aef7c5

add timer for files

aa20748

fix script

06f94fe

move script to file

7c38c1e

rename job

a1e4ea6

restart pipeline

811f7a6

add debug messages

b0498bd

enable script

0c7411a

restart pipeline to add second result to graph

6f7904e

remove debug refs

91b7386

alvicsam changed the title ~~[Do not merge] Publishing benchmarks for graphs~~ Publishing benchmarks for graphs Jul 25, 2023

alvicsam requested a review from Robbepop July 25, 2023 14:41

alvicsam requested a review from sergejparity July 25, 2023 14:47

Merge branch 'master' into as-bench

909863c

Robbepop reviewed Jul 29, 2023

View reviewed changes

Merge branch 'master' into as-bench

8cace27

Robbepop merged commit af8c588 into master Jul 31, 2023
17 checks passed

Robbepop deleted the as-bench branch July 31, 2023 10:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Publishing benchmarks for graphs #740

Publishing benchmarks for graphs #740

alvicsam commented Jul 24, 2023 •

edited

Loading

paritytech-cicd-pr commented Jul 24, 2023 •

edited

Loading

codecov-commenter commented Jul 25, 2023 •

edited

Loading

Robbepop commented Jul 26, 2023 •

edited

Loading

alvicsam commented Jul 27, 2023

Robbepop commented Jul 29, 2023

Robbepop left a comment

Robbepop Jul 29, 2023

sergejparity Jul 31, 2023

alvicsam Jul 31, 2023

Robbepop commented Jul 31, 2023

alvicsam commented Jul 31, 2023

Robbepop commented Jul 31, 2023

Publishing benchmarks for graphs #740

Publishing benchmarks for graphs #740

Conversation

alvicsam commented Jul 24, 2023 • edited Loading

paritytech-cicd-pr commented Jul 24, 2023 • edited Loading

BENCHMARKS

codecov-commenter commented Jul 25, 2023 • edited Loading

Codecov Report

Robbepop commented Jul 26, 2023 • edited Loading

alvicsam commented Jul 27, 2023

Robbepop commented Jul 29, 2023

Robbepop left a comment

Choose a reason for hiding this comment

Robbepop Jul 29, 2023

Choose a reason for hiding this comment

sergejparity Jul 31, 2023

Choose a reason for hiding this comment

alvicsam Jul 31, 2023

Choose a reason for hiding this comment

Robbepop commented Jul 31, 2023

alvicsam commented Jul 31, 2023

Robbepop commented Jul 31, 2023

alvicsam commented Jul 24, 2023 •

edited

Loading

paritytech-cicd-pr commented Jul 24, 2023 •

edited

Loading

codecov-commenter commented Jul 25, 2023 •

edited

Loading

Robbepop commented Jul 26, 2023 •

edited

Loading