feat: add benchmarks and performance checks on PRs #881

ajnavarro · 2023-06-08T13:16:43Z

Description

Add benchmark monitoring to keep track of improved/decreased performance. We also added checks to run fast benchmarks on every PR.

it closes #689

Still a WIP:

Use a GitHub self-hosted runner to keep benchmark results consistent
Add a README.md inside the .benchmarks folder to help people add new benchmarks to graphs and PR checks.
Remove -short flag when generating the graphs

Contributors Checklist

Added new tests, or not needed, or not feasible
Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory
Updated the official documentation or not needed
No breaking changes were made, or a BREAKING CHANGE: xxx message was included in the description
Added references to related issues and PRs
Provided any useful hints for running manual tests

moul · 2023-06-08T16:43:51Z

.benchmarks/gobenchdata-checks.yml

+      benchmarks: []
+      diff: (current.NsPerOp - base.NsPerOp) / base.NsPerOp * 100
+      thresholds:
+        max: 10


what it means, and what happens if we hit this?

these are rules that will be executed when the benchmark-check.yml action is triggered on every pr. It will compare the diff between the operations with the previous execution, and if the diff is bigger than 10%, it will fail.

example check output report: https://github.com/ajnavarro/gno/actions/runs/5280312972/jobs/9553479785#step:4:63

.benchmarks/gobenchdata-checks.yml

moul · 2023-06-08T16:45:11Z

.benchmarks/gobenchdata-web.yml

+    charts:
+      - name: Type assertion benchmarks (gnovm)
+        benchmarks: [ 'TypeAssertion.' ]
+        package: github.com\/gnolang\/gno\/gnovm\/pkg\/gnolang
+      - name: Reflect benchmarks (gnovm)
+        benchmarks: [ 'Reflect.' ]
+        package: github.com\/gnolang\/gno\/gnovm\/pkg\/gnolang
+      - name: Equality benchmarks (gnovm)
+        benchmarks: [ '.Equality' ]
+        package: github.com\/gnolang\/gno\/gnovm\/pkg\/gnolang
+      - name: LoopyMain benchmarks (gnovm)
+        benchmarks: [ '.LoopyMain' ]
+        package: github.com\/gnolang\/gno\/gnovm\/pkg\/gnolang
+      - name: Mempool (tm2)
+        package: ^github.com/gnolang/gno/tm2/pkg/bft/mempool$
+      - name: State (tm2)
+        package: ^github.com/gnolang/gno/tm2/pkg/bft/state$
+      - name: Types (tm2)
+        package: ^github.com/gnolang/gno/tm2/pkg/bft/types$
+      - name: WAL (tm2)
+        package: ^github.com/gnolang/gno/tm2/pkg/bft/wal$
+      - name: Bcrypt (tm2)
+        package: ^github.com/gnolang/gno/tm2/pkg/crypto/bcrypt$
+      - name: ed25519 (tm2)
+        package: ^github.com/gnolang/gno/tm2/pkg/crypto/ed25519$
+      - name: Merkle (tm2)
+        package: ^github.com/gnolang/gno/tm2/pkg/crypto/merkle$
+      - name: secp256k1 (tm2)
+        package: ^github.com/gnolang/gno/tm2/pkg/crypto/secp256k1$
+      - name: iavl (tm2)
+        package: ^github.com/gnolang/gno/tm2/pkg/iavl.
+      - name: random (tm2)
+        package: ^github.com/gnolang/gno/tm2/pkg/random$
+      - name: auth (tm2)
+        package: ^github.com/gnolang/gno/tm2/pkg/sdk/auth$
+      - name: cache (tm2)
+        package: ^github.com/gnolang/gno/tm2/pkg/store/cache$


we can probably generate this dynamically

Even if they seem to be the same, I had to tweak every package to have good graphs on the generated dashboard. As an example, LoopyMain benchmark takes a lot more time compared with other benchmarks on the same package, making the output graph unreadable, so I split them into two sections, one with the LoopyMain benchmark alone, and the other with all the other benchmarks.

moul · 2023-06-08T16:46:06Z

.github/workflows/benchmark-check.yml

+    - uses: actions/setup-go@v2
+      with: { go-version: "1.20" }
+    - name: gobenchdata publish
+      run: go run go.bobheadxi.dev/gobenchdata@v1 action


can you add a go install step before; just to make sure that we've more comprehensible error handling and sense of speed for install and run.

moul · 2023-06-08T16:49:20Z

wow, that looks powerful, and I like how it replaces an infra with git 👍

moul · 2023-06-09T07:02:09Z

.github/workflows/benchmark-publish.yml

@@ -0,0 +1,23 @@
+name: benchmarks publish


can this be merged to have a single github action; maybe by merging the on: triggers and adding a if push + branch=master condition?

hmm, they are two different things, one is adding information about performance deltas on the PR, and the other is generating the dashboard with all the benchmark results. I can merge them if it clarifies what benchmark action does.

.github/workflows/benchmark-publish.yml

ajnavarro · 2023-06-15T14:35:58Z

Until I have a self-hosted runner managed by the company and a token with write permissions, I managed to do it on my own on a fork and using my home server to run the benchmarks:

https://ajnavarro.com/gno-benchmarks/

~~I'm having some problems with the second run. I opened an issue: bobheadxi/gobenchdata#76~~ Problem fixed on my side, I was not using a stateless self-hosted runner

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

demand too Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

ajnavarro · 2023-06-16T10:21:07Z

Everything is tested and working on my side using a fork, we only need an access token with write permissions to write into benchmarks gh-pages branch all the results, and a stateless self-hosted runner to have consistent results every time we execute the tests.

CC @moul @albttx

zivkovicmilos

I like this effort very much - it's a start in the right direction for performance tracking

I've left minor comments, otherwise looks great 💯

tm2/pkg/std/coin_benchmark_test.go

tm2/pkg/iavl/benchmarks/bench_test.go

tm2/pkg/crypto/secp256k1/bench_test.go

tm2/pkg/bft/wal/wal_test.go

tm2/pkg/amino/benchmark_test.go

.github/pull_request_template.md

.github/workflows/benchmark.yml

zivkovicmilos · 2023-06-16T10:37:46Z

.benchmarks/README.md

@@ -0,0 +1,32 @@
+# Benchmarks
+
+This folder is where benchmarks are configured to be added on the dashboard generated in [benchmarks](https://gnoland.github.io/benchmarks).


@moul @albttx
Can we host these at benchmarks.gno.land, or somewhere similar?

good idea, let's do it

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

ajnavarro · 2023-06-19T10:05:34Z

As suggested by @moul , I changed to publish all benchmark results into a local branch, gh-benchmarks and added a process to copy that branch into benchmark's gh-pages branch here: gnolang/benchmarks#7

github-actions bot assigned ajnavarro Jun 8, 2023

github-actions bot added 📦 🌐 tendermint v2 Issues or PRs tm2 related 📦 🤖 gnovm Issues or PRs gnovm related labels Jun 8, 2023

ajnavarro changed the title ~~feature: add benchmarks and performance checks on PRs~~ feat: add benchmarks and performance checks on PRs Jun 8, 2023

ajnavarro force-pushed the feature/benchmarks branch from 6348a05 to 97e4e45 Compare June 8, 2023 13:50

moul reviewed Jun 8, 2023

View reviewed changes

.benchmarks/gobenchdata-checks.yml Outdated Show resolved Hide resolved

moul reviewed Jun 8, 2023

View reviewed changes

moul reviewed Jun 9, 2023

View reviewed changes

.github/workflows/benchmark-publish.yml Outdated Show resolved Hide resolved

ajnavarro force-pushed the feature/benchmarks branch 7 times, most recently from 0eacec2 to 3df7b56 Compare June 15, 2023 11:50

ajnavarro added the 🌟 improvement performance improvements, refactors ... label Jun 16, 2023

ajnavarro added 9 commits June 16, 2023 12:16

feature: add benchmarks and performance checks on PRs

bd745ca

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

Fix workflows

42ef1f7

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

Add requested changes

f379a45

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

test publish

75732b8

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

Fix go version

17347cd

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

Add readme to explain how to add new benchmarks on charts.

259f7ff

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

Fix chart groups and add checks on PRs

4a63e96

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

Improve benchmarks checks config

0a1b0dd

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

Test everything on forks

a10deb1

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

ajnavarro added 6 commits June 16, 2023 12:16

Change to self-hosted runner

8c1d017

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

Add workflow_dispatch to benchmarks to make possible to run them on

f159efb

demand too Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

Use special token to be able to write into benchmarks repo

89f8489

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

Move web config to benchmark repository

6e4c15a

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

Check without self-hosted runner

39ffa27

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

Fixed all long-running benchmarks

ef36bbd

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

ajnavarro force-pushed the feature/benchmarks branch from fc10691 to ef36bbd Compare June 16, 2023 10:16

ajnavarro marked this pull request as ready for review June 16, 2023 10:17

ajnavarro requested review from jaekwon and a team as code owners June 16, 2023 10:17

zivkovicmilos approved these changes Jun 16, 2023

View reviewed changes

zivkovicmilos requested a review from albttx June 16, 2023 10:39

ajnavarro added 3 commits June 16, 2023 16:01

Requested changes.

cdde9fb

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

Refactor benchmarks.yml to push to local branch

0709a0a

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

Improve benchmarks readme

19124cc

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>

moul approved these changes Jun 20, 2023

View reviewed changes

moul merged commit 60bab86 into gnolang:master Jun 20, 2023
72 of 73 checks passed

michelleellen mentioned this pull request Jul 12, 2023

Agenda (12.07.2023)_#9.md gnolang/meetings#17

Merged

10 tasks

michelleellen mentioned this pull request Jul 26, 2023

Create Agenda (26.07.2023)_#10.md gnolang/meetings#20

Closed

14 tasks

ajnavarro mentioned this pull request Jul 31, 2023

chore: more concise PR template #998

Merged

7 tasks

michelleellen mentioned this pull request Aug 8, 2023

Gno.land Developer Call Agenda {09.08.2023}_ #11 gnolang/meetings#25

Closed

16 tasks

Doozers pushed a commit to Doozers/gno that referenced this pull request Aug 31, 2023

feat: add benchmarks and performance checks on PRs (gnolang#881)

2267b21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add benchmarks and performance checks on PRs #881

feat: add benchmarks and performance checks on PRs #881

ajnavarro commented Jun 8, 2023 •

edited

moul Jun 8, 2023

ajnavarro Jun 12, 2023

ajnavarro Jun 16, 2023

moul Jun 8, 2023

ajnavarro Jun 12, 2023

moul Jun 8, 2023

moul commented Jun 8, 2023

moul Jun 9, 2023

ajnavarro Jun 12, 2023

ajnavarro commented Jun 15, 2023 •

edited

ajnavarro commented Jun 16, 2023

zivkovicmilos left a comment

zivkovicmilos Jun 16, 2023

moul Jun 20, 2023

ajnavarro commented Jun 19, 2023

		@@ -0,0 +1,32 @@
		# Benchmarks

		This folder is where benchmarks are configured to be added on the dashboard generated in [benchmarks](https://gnoland.github.io/benchmarks).

feat: add benchmarks and performance checks on PRs #881

feat: add benchmarks and performance checks on PRs #881

Conversation

ajnavarro commented Jun 8, 2023 • edited

Description

Contributors Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moul commented Jun 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajnavarro commented Jun 15, 2023 • edited

ajnavarro commented Jun 16, 2023

zivkovicmilos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajnavarro commented Jun 19, 2023

ajnavarro commented Jun 8, 2023 •

edited

ajnavarro commented Jun 15, 2023 •

edited