ci: add github workflow for performance regression check #95

SilverRainZ · 2021-10-09T12:11:49Z

This PR add a github workflow for performance regression check (using benchdiff, a wrapper of benchstat).

Close #28.

Usage

The workflow will be triggered:

when a PR is created, or
anyone comments "/benchdiff" on PR

It takes a few minutes to run (the time is depends on the go packages modified by this PR), once it is finished, benchdiff result will be posted on this PR:

Non-full amount

This workflow won't do a full amount benchmarking, it only do bench on modified go packages (get by git diff <default_branch> <head_of_pr>, for details, refer to jobs.benchdiff.steps.head). In the aboved case, only packages collection/skipset and collection/skipmap are modified. If there is no go package modified, the following comment will be posted:

Github Checks

This workflow also creates a check for the HEAD commit of PR.

The details of check looks like this:

luchsh · 2021-10-11T03:09:01Z

/benchdiff

SilverRainZ · 2021-10-11T03:11:32Z

/benchdiff

It takes effect after this PR is merged to default branch (develop).

.github/workflows/pr-benchdiff.yml

luchsh · 2021-10-11T03:15:30Z

/benchdiff

it does not work for this issue :(

SilverRainZ · 2021-10-11T03:19:45Z

it does not work for this issue :(

Yes, see my comment aboved. You can try this PR: SilverRainZ#3

luchsh · 2021-10-11T03:33:26Z

it does not work for this issue :(

Yes, see my comment aboved. You can try this PR: SilverRainZ#3

I'm trying to trigger the action multiple times by keep commenting '/benchdiff', let's see if the results are stable enough. (I think this is crucial for a performance-oriented action)

SilverRainZ · 2021-10-11T09:00:17Z

@luchsh @zhangyunhao116 @PureWhiteWu This PR is ready for review, please take a look.

zhangyunhao116 · 2021-10-11T09:53:58Z

.github/workflows/pr-benchdiff.yml

+      - name: Setup go
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.16


Maybe we could setup the os by os: [ubuntu-latest] ? I wonder if we can also use something like go-latest instead of 1.16 ?

It seems that action/setup-go supports semver, ^1.x should the the thing we need.

This PR use 1.16 because it is used in all of other workflows. If we use ^1.x, it would be better to update all of them.

$ grep -r go-version codeql-analysis.yml: go-version: 1.16 push-check.yml: go-version: 1.16 pr-benchdiff.yml: go-version: 1.16 pr-check.yml: go-version: 1.16

zhangyunhao116 · 2021-10-11T09:57:28Z

.github/workflows/pr-benchdiff.yml

+            --packages="${{ steps.head.outputs.pkgs }}"
+            --count=10
+            --warmup-count=1
+            --benchtime=100000x


The benchtime may not be suitable for all packages, some packages will take a long time for each bench, it is configurable?

It may not be easy, --benchtime take effects for all --packages, and both them are action-level configuration, it is not easy to run a action for every single package.

If we can make sure that we changes at most one package in a PR, we can create a mapping from package name to --benchtime (even more bench configuration), thing will be easier.

Is there any bench configuraion suitable for all packages? per-package configuration is not easy for now.

Maybe no... but benchtime=1s is ok for ~70% packages

OK, let's use benchtime=1s at least in this PR.

Comment "/benchdiff" to trigger this check

- support "pull_reqeust" event - filter deleted go packages - post a comment on job started - post a comment on job failed

@zhangyunhao116

* release: 20210913 (#83) (#88) * feat: circuitbreaker.panel use skipmap (#78) * fix(metainfo): fix misuse of append (#79) * fix(lscq): add write barrier for LSCQ Pointer (#80) * feat(metainfo): improve backward APIs (#81) * release: 20210913 (#83) (#93) * feat: circuitbreaker.panel use skipmap (#78) * fix(metainfo): fix misuse of append (#79) * fix(lscq): add write barrier for LSCQ Pointer (#80) * feat(metainfo): improve backward APIs (#81) * chore(skipmap,skipset): remove duplicated code generation declaration (#87) * fix(lscq): cas2 use runtime.noescape (#94) * feat: add fastrand.Read() (#90) Co-authored-by: liyichao <liyichao@bytedance.com> * doc: add badge to link to godoc (#99) * ci: skip golang related workflows when there are only doc changes (#101) * ci: add workflow for feishu/lark notification (#100) * ci: add workflow for feishu/lark notification * ci: fix typo * ci: also send feishu notification on issue opened Co-authored-by: Jonathan Lu <luchsh@users.noreply.github.com> * feat(metainfo): define standard for backward prefix (#102) * feat(fastrand): support Read,Shuffle,Perm (#103) * feat(fastrand): Read remove temp buffer (#104) * ci: add github workflow for performance regression check (#95) * ci: add github workflow for performance regression check Comment "/benchdiff" to trigger this check * ci: add comments to pr-benchdiff.yml * ci: report benchdiff result to PR_HEAD's check run * ci: various changes - support "pull_reqeust" event - filter deleted go packages - post a comment on job started - post a comment on job failed * ci: let xargs ignore empty line * ci: skip benchdiff when there are only doc changes * ci: set -x for debugging * ci: make sure we are using github default branch as baseline * ci: set benchtime=1s for benchdiff * chore: add github issue forms (#105) * test(lang/syncx): add benchmark for RWMutex (#89) * fix(xxhash3): add fallback to fix panic occurs on non avx2, non sse2 machine (#108) Co-authored-by: Ye Li <liye.e@bytedance.com> * feat: add zset (#98) * feat: add zset * chore(zset): update comment * docs(zset): add readme * chore: some code style fixes * chore(zset): rename Float64RangeOpt -> RangeOpt * chore(zset): comment style fixes * chore(zset): add a todo about maxLevel * chore(zset): another comment fix * chore(zset): move skiplist impl to another file * chore(zset): add license header for opt.go * chore: add myself as zset's CODEOWNER * zset: don't use z.Range(0, z.Len()-1) * doc: remove redundant section * fix(zset): break when key is not exist * chore(zset): simplify func name * chore(zset): update cheatsheet * chore(zset): meaningful const * refactor(zset): optionalArray.init() * docs: update zset readme * docs(zset): also add @zhangyunhao116 as code owner * docs(zset): some grammar fixes * docs: fix docs typo (#109) * feat: auto tuning gc (#112) * feat: auto tuning gc * doc: gctuner * fix: gctuner tests data race (#113) * fix: gctuner tests data race * chore: add owner * chore(ci): use self-hosted runner (#114) * chore(ci): use self-hosted runner * fix lint Co-authored-by: Shengyu Zhang <reg@silverrainz.me> Co-authored-by: ziposcar <499581494@qq.com> Co-authored-by: liyichao <liyichao@bytedance.com> Co-authored-by: Shengyu Zhang <zhangshengyu.0@bytedance.com> Co-authored-by: Jonathan Lu <luchsh@users.noreply.github.com> Co-authored-by: Pure White <wudi.daniel@bytedance.com> Co-authored-by: lyeeeeee <awesomeyeli@gmail.com> Co-authored-by: Ye Li <liye.e@bytedance.com> Co-authored-by: chyroc <chyroc@qq.com> Co-authored-by: Joway <joway.w@gmail.com> Co-authored-by: Joway <wangzhuowei@bytedance.com>

luchsh reviewed Oct 11, 2021

View reviewed changes

.github/workflows/pr-benchdiff.yml Show resolved Hide resolved

.github/workflows/pr-benchdiff.yml Show resolved Hide resolved

luchsh closed this Oct 11, 2021

luchsh reopened this Oct 11, 2021

SilverRainZ marked this pull request as ready for review October 11, 2021 08:58

SilverRainZ requested review from PureWhiteWu and zhangyunhao116 as code owners October 11, 2021 08:58

zhangyunhao116 reviewed Oct 11, 2021

View reviewed changes

luchsh previously approved these changes Oct 11, 2021

View reviewed changes

luchsh mentioned this pull request Oct 13, 2021

feat: add zset #98

Merged

5 tasks

SilverRainZ dismissed luchsh’s stale review via 9f46190 October 14, 2021 03:00

SilverRainZ force-pushed the ci/benchdiff branch from 9f46190 to 31538e5 Compare October 14, 2021 03:13

luchsh previously approved these changes Oct 21, 2021

View reviewed changes

zhangyunhao116 previously approved these changes Oct 21, 2021

View reviewed changes

SilverRainZ enabled auto-merge (squash) October 21, 2021 08:06

SilverRainZ disabled auto-merge October 21, 2021 08:06

SilverRainZ added 9 commits October 21, 2021 16:08

ci: add github workflow for performance regression check

143f591

Comment "/benchdiff" to trigger this check

ci: add comments to pr-benchdiff.yml

f80922c

ci: report benchdiff result to PR_HEAD's check run

645d6ed

ci: various changes

c7af739

- support "pull_reqeust" event - filter deleted go packages - post a comment on job started - post a comment on job failed

ci: let xargs ignore empty line

b88c52a

ci: skip benchdiff when there are only doc changes

281e850

ci: set -x for debugging

832054b

ci: make sure we are using github default branch as baseline

ca9d49b

ci: set benchtime=1s for benchdiff

0ff6336

SilverRainZ dismissed stale reviews from zhangyunhao116 and luchsh via 0ff6336 October 21, 2021 08:08

SilverRainZ force-pushed the ci/benchdiff branch from c646116 to 0ff6336 Compare October 21, 2021 08:08

luchsh approved these changes Oct 21, 2021

View reviewed changes

zhangyunhao116 approved these changes Oct 21, 2021

View reviewed changes

SilverRainZ merged commit 9ad039f into develop Oct 21, 2021

SilverRainZ deleted the ci/benchdiff branch October 21, 2021 08:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: add github workflow for performance regression check #95

ci: add github workflow for performance regression check #95

SilverRainZ commented Oct 9, 2021 •

edited

Loading

luchsh commented Oct 11, 2021

SilverRainZ commented Oct 11, 2021

luchsh commented Oct 11, 2021

SilverRainZ commented Oct 11, 2021

luchsh commented Oct 11, 2021 •

edited

Loading

SilverRainZ commented Oct 11, 2021

zhangyunhao116 Oct 11, 2021

SilverRainZ Oct 11, 2021

zhangyunhao116 Oct 11, 2021

SilverRainZ Oct 11, 2021

SilverRainZ Oct 20, 2021

zhangyunhao116 Oct 20, 2021

SilverRainZ Oct 21, 2021

SilverRainZ Oct 21, 2021

ci: add github workflow for performance regression check #95

ci: add github workflow for performance regression check #95

Conversation

SilverRainZ commented Oct 9, 2021 • edited Loading

Usage

Non-full amount

Github Checks

luchsh commented Oct 11, 2021

SilverRainZ commented Oct 11, 2021

luchsh commented Oct 11, 2021

SilverRainZ commented Oct 11, 2021

luchsh commented Oct 11, 2021 • edited Loading

SilverRainZ commented Oct 11, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SilverRainZ commented Oct 9, 2021 •

edited

Loading

luchsh commented Oct 11, 2021 •

edited

Loading