Skip to content

knqyf263/cob

master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
img
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

GitHub release Go Report Card MIT License

Abstract

cob compares benchmarks between the latest commit (HEAD) and the previous commit (HEAD{@1}). The program will fail if the change in score is worse than the threshold. This tools is suitable for CI/CD to detect a regression of a performance automatically.

cob runs go test -bench before and after commit internally, so it depends on go command.

CAUTION: Note that git reset is executed when you run cob. You should commit all changes before running cob.

Table of Contents

Continuous Integration (CI)

See cob-example for details.

GitHub Actions

name: Bench
on: [push, pull_request]
jobs:
  test:
    name: Bench
    runs-on: ubuntu-latest
    steps:

    - name: Set up Go 1.13
      uses: actions/setup-go@v1
      with:
        go-version: 1.13
      id: go

    - name: Check out code into the Go module directory
      uses: actions/checkout@v1

    - name: Install cob
      run: curl -sfL https://raw.githubusercontent.com/knqyf263/cob/master/install.sh | sudo sh -s -- -b /usr/local/bin

    - name: Run Benchmark
      run: cob

Travis CI

dist: bionic
language: go
go:
  - 1.13.x

before_script:
  - curl -sfL https://raw.githubusercontent.com/knqyf263/cob/master/install.sh | sudo sh -s -- -b /usr/local/bin

script:
  - cob

CircleCI

version: 2
jobs:
  bench:
    docker:
      - image: circleci/golang:1.13
    steps:
      - checkout
      - run:
          name: Install cob
          command: curl -sfL https://raw.githubusercontent.com/knqyf263/cob/master/install.sh | sudo sh -s -- -b /usr/local/bin
      - run:
          name: Run cob
          command: cob
workflows:
  version: 2
  build-workflow:
    jobs:
      - bench

Example

Override a command to measure benchmarks

To measure benchmarks by make bench, you can use -bench-cmd and -bench-args options.

$ cob -bench-cmd make -bench-args bench

Run only those benchmarks matching a regular expression

$ cob -bench-args "test -bench Append -benchmem ./..."
Result
2020/01/12 17:32:30 Run Benchmark: 4363944cbed3da7a8245cbcdc8d8240b8976eb24 HEAD{@1}
2020/01/12 17:32:32 Run Benchmark: 599a5523729d4d99a331b9d3f71dde9e1e6daef0 HEAD

Result
======

+-----------------------------+----------+---------------+-------------------+
|            Name             |  Commit  |    NsPerOp    | AllocedBytesPerOp |
+-----------------------------+----------+---------------+-------------------+
| BenchmarkAppend_Allocate-16 |   HEAD   |  179.00 ns/op |      117 B/op     |
+                             +----------+---------------+-------------------+
|                             | HEAD@{1} |  115.00 ns/op |      23 B/op      |
+-----------------------------+----------+---------------+-------------------+

Comparison
==========

+-----------------------------+---------+-------------------+
|            Name             | NsPerOp | AllocedBytesPerOp |
+-----------------------------+---------+-------------------+
| BenchmarkAppend_Allocate-16 | 55.65%  |      408.70%      |
+-----------------------------+---------+-------------------+

Show only benchmarks with worse score

$ cob -only-degression
Result
2020/01/12 17:48:35 Run Benchmark: 4363944cbed3da7a8245cbcdc8d8240b8976eb24 HEAD{@1}
2020/01/12 17:48:38 Run Benchmark: 599a5523729d4d99a331b9d3f71dde9e1e6daef0 HEAD

Comparison
==========

+-----------------------------+---------+-------------------+
|            Name             | NsPerOp | AllocedBytesPerOp |
+-----------------------------+---------+-------------------+
| BenchmarkAppend_Allocate-16 | 52.34%  |      347.83%      |
+-----------------------------+---------+-------------------+

2020/01/12 17:48:39 This commit makes benchmarks worse

Specify a threshold

The following option means the program fails if a benchmark score gets worse than 50%.

$ cob -threshold 0.5 ./...

Specify a base commit compared with HEAD

By default, cob uses HEAD~1. If you compare benchmarks with different commit, you can use --base option.

$ cob --base origin/master ./...

Compare only memory allocation

You can use -compare option.

$ cob -compare B/op
Result
2020/01/15 14:46:31 Run Benchmark: 4363944cbed3da7a8245cbcdc8d8240b8976eb24 HEAD~1
2020/01/15 14:46:33 Run Benchmark: 599a5523729d4d99a331b9d3f71dde9e1e6daef0 HEAD

Result
======

+-----------------------------+----------+---------------+-------------------+
|            Name             |  Commit  |    NsPerOp    | AllocedBytesPerOp |
+-----------------------------+----------+---------------+-------------------+
| BenchmarkAppend_Allocate-16 |   HEAD   |  179.00 ns/op |      121 B/op     |
+                             +----------+---------------+-------------------+
|                             | HEAD@{1} |  104.00 ns/op |      23 B/op      |
+-----------------------------+----------+---------------+-------------------+
|      BenchmarkCall-16       |   HEAD   |   0.50 ns/op  |       0 B/op      |
+                             +----------+---------------+                   +
|                             | HEAD@{1} |   0.49 ns/op  |                   |
+-----------------------------+----------+---------------+-------------------+

Comparison
==========

+-----------------------------+---------+-------------------+
|            Name             | NsPerOp | AllocedBytesPerOp |
+-----------------------------+---------+-------------------+
| BenchmarkAppend_Allocate-16 |    -    |      426.09%      |
+-----------------------------+---------+-------------------+
|      BenchmarkCall-16       |    -    |       0.00%       |
+-----------------------------+---------+-------------------+

2020/01/15 14:46:35 This commit makes benchmarks worse

Skip running cob

If your commit message contains [skip cob], cob is skipped.

$ git add README.md
$ git commit -m "[skip cob] update README.md"
$ cob
2020/04/19 12:46:57 [skip cob] is detected, so the benchmark is skipped

Usage

NAME:
   cob - Continuous Benchmark for Go project

USAGE:
   cob [global options] command [command options] [arguments...]

COMMANDS:
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --only-degression   Show only benchmarks with worse score (default: false)
   --threshold value   The program fails if the benchmark gets worse than the threshold (default: 0.2)
   --base value        Specify a base commit compared with HEAD (default: "HEAD~1")
   --compare value     Which score to compare (default: "ns/op,B/op")
   --bench-cmd value   Specify a command to measure benchmarks (default: "go")
   --bench-args value  Specify arguments passed to -cmd (default: "test -run '^$' -bench . -benchmem ./...")
   --help, -h          show help (default: false)

Q&A

Benchmarks with the same name

Specify a package name.

$ cob -bench-args "test -bench . -benchmem ./foo" 
$ cob -bench-args "test -bench . -benchmem ./bar" 

A result of benchmarks is unstable

You can specify -benchtime.

$ cob -bench-args "test -bench . -benchmem -benchtime 10s ./..." 

License

This repository is available under the MIT

Author

Teppei Fukuda (knqyf263)