Go Bench Away!

Utility to benchmark any Golang project and visualize results

As of this writing, the project is maturing from "proof of concept" to "open-source utility". Some things may be broken, some are going to break. Also, no documentations and no tests.

Before V1.0:

• Handle re-delivery (call InProgress() ?)
• Add wait option to submit
• Submit multiple
• Documentation and examples
• Split template to reuse style and other common elements
• Tests
• Less code repetition in commands

• Spin loop in worker when server is down

Future/Wishlist:

• Embedded mode - run server and worker in-process
• Expose internal as packages so parts can be used as library
• Search jobs
• Missing datapoint could be NaN rather than zeroes (tested in Trend, probably true in other plots)
• Fetch job records in parallel

---

What is this?

go-bench-away (GBA) is a utility to orchestrate and automate execution and analysis of benchmarks. It is inspired by tools such as Jenkins and Genie.

Unlike those generic tools, GBA does just one thing: run benchmarks, produce results visual reports.

GBA works with any Golang repository that implements testing.Benchmark, without any change to the target repository.

It is also similar to gobenchdata, except is designed to natively run benchmarks elsewhere (i.e., not on your laptop, but on a dedicated bare metal host somewhere in the cloud), hence the name go-bench-away.

Installation

Download the latest release or install from source with:

$ go install github.com/mprimi/go-bench-away@vX.Y.Z

NATS Server setup

$ nats-server -js -auth ${TOKEN}

Schema initialization

One-time initialization (creates empty stream, key-value store, and object store):

$ go-bench-away -server nats://${TOKEN}@${SERVER_IP}:4222 init
2022/10/20 19:56:54 Jobs stream: default-jobs
2022/10/20 19:56:54 Jobs KV store created: default-jobs
2022/10/20 19:56:54 Artifacts Obj store: default-artifacts

Worker setup

On the benchmark host (i.e., "bare metal" host where benchmarks will execute)

$ go-bench-away -server nats://${TOKEN}@${SERVER_IP}:4222 worker
2022/10/20 19:57:04 Worker ready

This is a long-running process, so you may want to run it inside a screen session, or as a daemon service

Submit a job

Run this from anywhere: your laptop, a GitHub action, a Jenkins job, etc.

$ go-bench-away -server nats://${TOKEN}@${SERVER_IP}:4222 submit -remote https://github.com/nats-io/nats-server.git -ref v2.9.3 -reps 3 -tests_dir server -filter 'BenchmarkJetStreamPublish/.*/Sync'

Reference

Testing different Go versions

It is possible to test the same source using different Go versions by passing different -go_path flag to the submit command. Example:

# Helper function to simplify installation
# (may need to run as root depending on install location )
function install_go () {
  local goversion="${1}"
  local goos="${2}"
  local goarch="${3}"
  local install_dir="${4}/go${goversion}"

  local archive_name="go${goversion}.${goos}-${goarch}.tar.gz"
  local download_url="https://go.dev/dl/${archive_name}"
  local tempdir="/tmp/go${goversion}"

  echo "Installing: Go ${goversion} (${goos} ${goarch}) in ${install_dir}"

  mkdir "${tempdir}"
  curl --silent -L "${download_url}" -o "${tempdir}/${archive_name}"
  tar -xf "${tempdir}/${archive_name}" -C ${tempdir}
  mv ${tempdir}/go ${install_dir}

  echo "Installed: ${install_dir}/bin/go"
  "${install_dir}/bin/go" version
}

# Install some set of versions
for go_version in 1.19.3 1.18.8 1.17.13 1.16.15 1.15.15 1.14.15;
do
  install_go ${go_version} linux amd64 /usr/local
done

Then submit jobs (from anywhere):

for go_version in 1.19.3 1.18.8;
do
  go-bench-away -server [...] submit [...] -go_path "/usr/local/go${go_version}"
done