From 94b6bcaa71c5c3aba685c233eb590f12b1ccde0c Mon Sep 17 00:00:00 2001 From: Gleb Alekseev Date: Sat, 8 Aug 2026 19:09:00 +0200 Subject: [PATCH 1/2] Compare two revisions on the host, without Docker A container on macOS is a Linux virtual machine and the Apple GPU is not passed into it, so the benchmark image cannot measure one: there is no counterpart to the NVIDIA Container Toolkit, and the image registers the NVIDIA ICD in any case. Building the image for linux/arm64 would change the CPU architecture and nothing else. prepare-native.sh does on the host what the src and build stages of the Dockerfile do inside the image - it resolves both revisions, exports each one with git archive, runs make in it, and leaves the layout run-benchmark.sh expects - so the same runner serves both paths. It reads the local repository by default, since the commits are usually already there and an export keeps the working tree out of the build. Three things kept the runner from working outside the image: /opt/bench was hardcoded, macOS ships bash 3.2 where `local -A` and ${slot^^} do not exist, and the OpenCL check relies on clinfo, which macOS does not have. A Darwin host without clinfo is now treated as having one platform, leaving the real check to profanity2's own device enumeration. Nothing here is macOS-only: the same two commands compare two revisions on any host with a working OpenCL runtime. Co-authored-by: Cursor --- bench/README.md | 38 +++++++ bench/prepare-native.sh | 219 ++++++++++++++++++++++++++++++++++++++++ bench/run-benchmark.sh | 58 ++++++++--- 3 files changed, 300 insertions(+), 15 deletions(-) create mode 100755 bench/prepare-native.sh diff --git a/bench/README.md b/bench/README.md index bd56845..933259c 100644 --- a/bench/README.md +++ b/bench/README.md @@ -2,6 +2,8 @@ This directory builds a **separate image** from the one in the repository root. The image in the root ships profanity2; this one measures it. It contains two revisions of profanity2 side by side and runs them alternately on the same GPU, so a change can be judged without trusting that two rented machines are equally fast. +There are two ways to put the two revisions side by side. [bench/build.sh](build.sh) bakes them into an image for a rented NVIDIA GPU, which is the rest of this document; [bench/prepare-native.sh](prepare-native.sh) builds them on the machine you are sitting at, which is the only way to measure an Apple GPU - see [Running natively](#running-natively-no-docker). Both then hand over to the same [run-benchmark.sh](run-benchmark.sh). + The result of a run looks like this: ``` @@ -83,6 +85,8 @@ Both revisions are compiled inside the image from a fresh clone, so your working bench/build.sh master pr/57 --repo https://github.com/YOUR_USER/profanity2 ``` +A native build resolves the same set of revisions, but reads the local repository rather than a clone, so a local branch works as a revision too and `--repo` is only needed for a fork or for pull request refs you never fetched. + ## Running it ```bash @@ -98,6 +102,39 @@ docker run --rm --gpus all profanity2-bench:master__pr-57 The image is built for `linux/amd64` by default because the Linux branch of the Makefile passes `-mmmx` and `-mcmodel=large`, which do not exist on arm64, and because GPU rental platforms are x86_64 anyway. On Apple Silicon the build therefore runs under emulation and takes a few minutes. +## Running natively (no Docker) + +On macOS a container is a Linux virtual machine, and the Apple GPU is not passed into it: there is no counterpart to the NVIDIA Container Toolkit, the Ubuntu base image cannot reach `OpenCL.framework` on the host, and this directory's Dockerfile registers the NVIDIA ICD in any case. Rebuilding the image for `linux/arm64` changes the CPU architecture and nothing else, so a container on a Mac measures at best a CPU OpenCL runtime. To compare two revisions on an Apple GPU, build them on the host: + +```bash +WORK=$(bench/prepare-native.sh 9011bcd pr57-head) +BENCH_ROOT=$WORK bench/run-benchmark.sh --mode leading --repeats 3 +``` + +[prepare-native.sh](prepare-native.sh) does on the host what the `src` and `build` stages of the Dockerfile do inside the image: it resolves both revisions, exports each one with `git archive`, runs `make` in it, and leaves the layout the runner expects. + +``` +/a/{profanity2.x64,keccak.cl,profanity.cl} +/b/{profanity2.x64,keccak.cl,profanity.cl} +/{a,b}.ref, /{a,b}.sha, /timer.state +``` + +By default the revisions come from **the local repository, used as is** - a commit you already have needs no network, and an export rather than a checkout keeps your working tree and its stale object files out of the build. `--repo ` clones instead, which is what a `pr/` revision needs when its refs were never fetched. The workdir sits under `TMPDIR` and is named after the two commits, so rerunning the same pair skips the compilation; `--force` rebuilds anyway. + +The runner is the same script the image runs, so everything under [Options](#options) and [Getting a number that means something](#getting-a-number-that-means-something) applies unchanged. It only needs `BENCH_ROOT`, which is the one thing `prepare-native.sh` prints on stdout. + +Nothing here is macOS-only: on a Linux box with a working OpenCL runtime the same two commands compare two revisions without building an image. + +### What to watch for on a Mac + +**`clinfo` is usually not installed**, and macOS has no ICD loader to interrogate. The runner treats a Darwin host without `clinfo` as having one platform and lets profanity2's own device enumeration be the real check - a machine with no usable device fails at the first run with its output printed. Install `clinfo` from Homebrew if you want the check back. + +**Thermal throttling is the thing most likely to ruin the result.** A laptop that heats up during the second half of a comparison hands the penalty to whichever revision ran last, which is exactly what the alternating A B A B order and the reported `spread` exist to expose. Keep the machine plugged in, and if the spread comes out anywhere near the difference between the revisions, the runner says so and the answer is more `--repeats` or a longer `--seconds`. + +**Check the work size once.** The default `-w 64` is accepted by an M4 Max, but on a GPU that rejects it profanity2 prints `warning: local work size abandoned on GPU0` and lets the driver choose. That is not fatal, but it has to happen for both revisions or the comparison is between two different work sizes - the warning goes to stdout, so a short run with `--seconds 20 --warmup 10 --repeats 1` is the cheapest way to see it before committing to the full run. + +**The MH/s figure is not comparable to the NVIDIA table** in the root [README](../README.md#benchmarks---current-version). What a native run gives you is A against B on one machine. + ## Options Everything has a flag and an environment variable; flags are easier on platforms that pass arguments to the entrypoint, variables are the only option on platforms that replace it. @@ -112,6 +149,7 @@ Everything has a flag and an environment variable; flags are easier on platforms | `--mask` | `BENCH_EXACT_MASK` | `deadbee` | mask for `--mode exact`, 4 to 10 fixed hex characters | | `--public-key` | `BENCH_PUBLIC_KEY` | secp256k1 generator | seed public key | | | `BENCH_SKIP_GPU_CHECK` | unset | start even when no OpenCL platform is detected | +| | `BENCH_ROOT` | `/opt/bench` | directory holding the two revisions, for a native run | A plain `PUBLIC_KEY` is honoured as well, but only when it holds 128 hexadecimal characters. Rental platforms hand out that generic name for their own SSH key, and it may already sit in your account-wide environment variables, so a value that is not a seed public key is ignored and the run falls back to the default. The header of every run says which key it used: diff --git a/bench/prepare-native.sh b/bench/prepare-native.sh new file mode 100755 index 0000000..11888fb --- /dev/null +++ b/bench/prepare-native.sh @@ -0,0 +1,219 @@ +#!/usr/bin/env bash +# +# Builds two revisions of profanity2 on this machine so that their speed can be +# compared without Docker. It does on the host what the src and build stages of +# bench/Dockerfile do inside the image, and lays the result out the way +# bench/run-benchmark.sh expects: +# +# /a/{profanity2.x64,keccak.cl,profanity.cl} +# /b/{profanity2.x64,keccak.cl,profanity.cl} +# /{a,b}.ref, /{a,b}.sha, /timer.state +# +# This is the only way to measure an Apple GPU: a container on macOS is a Linux +# virtual machine with no access to it. +# +# usage: bench/prepare-native.sh REF_A REF_B [OPTIONS] +# +# REF_A, REF_B anything the repository can resolve: a branch, a +# tag, a SHA, or pr/ for a pull request head +# --repo repository to take the revisions from, cloned into a +# cache directory [default = the repository this +# script lives in, used as is] +# --workdir where to build [default = a directory under TMPDIR +# named after the two commits] +# --force rebuild even when the binaries are already there +# -h, --help this text +# +# The workdir is printed on the last line, so the two steps compose: +# +# WORK=$(bench/prepare-native.sh 9011bcd pr57-head) +# BENCH_ROOT=$WORK bench/run-benchmark.sh + +set -euo pipefail + +REPO="${BENCH_REPO:-}" +WORKDIR="${BENCH_WORKDIR:-}" +FORCE=0 +REF_A="" +REF_B="" + +tmp_root="${TMPDIR:-/tmp}" +readonly CACHE_ROOT="${tmp_root%/}/profanity2-bench" + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +usage() { + sed -n '/^# usage:/,/^$/p' "${BASH_SOURCE[0]}" | sed 's/^#\{0,1\} \{0,1\}//' +} + +log() { + printf 'prepare-native.sh: %s\n' "$*" >&2 +} + +die() { + printf 'prepare-native.sh: error: %s\n' "$*" >&2 + exit 1 +} + +sanitize() { + printf '%s' "$1" | tr -c 'A-Za-z0-9_.-' '-' +} + +while [ "$#" -gt 0 ]; do + case "$1" in + --repo) REPO="${2:?--repo needs a value}"; shift 2 ;; + --workdir) WORKDIR="${2:?--workdir needs a value}"; shift 2 ;; + --force) FORCE=1; shift ;; + -h|--help) usage; exit 0 ;; + -*) die "unknown option '$1', try --help" ;; + *) + if [ -z "$REF_A" ]; then + REF_A="$1" + elif [ -z "$REF_B" ]; then + REF_B="$1" + else + die "expected exactly two revisions, got a third: '$1'" + fi + shift + ;; + esac +done + +if [ -z "$REF_A" ] || [ -z "$REF_B" ]; then + die "two revisions are required, try --help" +fi + +command -v git >/dev/null 2>&1 || die "git is not installed" +command -v make >/dev/null 2>&1 || die "make is not installed - run xcode-select --install on macOS" + +# --------------------------------------------------------------------------- +# Where the revisions come from +# --------------------------------------------------------------------------- +# Without --repo the local repository is used directly: the commits are almost +# always already there, and nothing has to be downloaded. Only a clone is ever +# fetched into, so the user's own repository is never modified. +if [ -n "$REPO" ]; then + SRC_REPO="$CACHE_ROOT/clones/$(sanitize "$REPO")" + if git -C "$SRC_REPO" rev-parse --git-dir >/dev/null 2>&1; then + log "updating the clone of $REPO" + git -C "$SRC_REPO" fetch --quiet --tags origin || die "cannot fetch from $REPO" + else + log "cloning $REPO" + mkdir -p "$(dirname "$SRC_REPO")" + git clone --quiet --no-checkout "$REPO" "$SRC_REPO" || die "cannot clone $REPO" + fi + # Pull request heads live outside refs/heads and have to be asked for by + # name, otherwise a commit that exists only in a pull request cannot be + # resolved. + git -C "$SRC_REPO" fetch --quiet origin '+refs/pull/*/head:refs/remotes/origin/pr/*' \ + || log "warning: could not fetch pull request refs from $REPO" +else + SRC_REPO="$(git -C "$script_dir" rev-parse --show-toplevel 2>/dev/null)" \ + || die "$script_dir is not inside a git repository, pass --repo" +fi + +resolve_ref() { + local ref="$1" + local candidate sha + + for candidate in "$ref" "origin/$ref" "refs/pull/${ref#pr/}/head"; do + sha="$(git -C "$SRC_REPO" rev-parse --verify --quiet "${candidate}^{commit}" 2>/dev/null || true)" + if [ -n "$sha" ]; then + printf '%s\n' "$sha" + return 0 + fi + done + + return 1 +} + +die_unresolved() { + local ref="$1" + + log "error: cannot resolve revision '$ref' in $SRC_REPO" + case "$ref" in + pr/*|pull/*) + log " a pull request head has to be fetched before it can be used:" + log " git -C $SRC_REPO fetch origin '+refs/pull/*/head:refs/remotes/origin/pr/*'" + log " or take both revisions from a clone: --repo " + ;; + *) + log " a branch, tag or commit that this repository does not have" + log " has to come from a clone: --repo " + ;; + esac + exit 1 +} + +SHA_A="$(resolve_ref "$REF_A")" || die_unresolved "$REF_A" +SHA_B="$(resolve_ref "$REF_B")" || die_unresolved "$REF_B" + +SHORT_A="$(git -C "$SRC_REPO" rev-parse --short=7 "$SHA_A")" +SHORT_B="$(git -C "$SRC_REPO" rev-parse --short=7 "$SHA_B")" + +if [ "$SHA_A" = "$SHA_B" ]; then + die "both revisions resolve to $SHORT_A, there is nothing to compare" +fi + +# Naming the workdir after the commits rather than the refs means a branch that +# moved gets a fresh directory instead of quietly reusing yesterday's build, +# and that rerunning the same pair skips the compilation. +if [ -z "$WORKDIR" ]; then + WORKDIR="$CACHE_ROOT/${SHORT_A}__${SHORT_B}" +fi + +log "A = $REF_A ($SHORT_A)" +log "B = $REF_B ($SHORT_B)" +log "repository = $SRC_REPO" +log "workdir = $WORKDIR" + +jobs_count="$(sysctl -n hw.ncpu 2>/dev/null || getconf _NPROCESSORS_ONLN 2>/dev/null || echo 4)" + +# Each revision gets its own directory because profanity2 reads keccak.cl and +# profanity.cl from the working directory and caches the compiled kernel there +# as cache-opencl.*; sharing one directory would have the revisions run each +# other's kernel. +build_slot() { + local slot="$1" ref="$2" sha="$3" short="$4" + local src="$WORKDIR/src-$slot" dst="$WORKDIR/$slot" + + if [ "$FORCE" -eq 0 ] && [ -x "$dst/profanity2.x64" ]; then + log "$slot: reusing the existing build of $short" + else + log "$slot: exporting $short" + rm -rf "$src" "$dst" + mkdir -p "$src" "$dst" + # An export rather than a checkout, so that the working tree, its + # stale object files and any uncommitted change stay out of the + # build. + git -C "$SRC_REPO" archive --format=tar "$sha" | tar -x -C "$src" + + log "$slot: building $short with make -j$jobs_count" + make -C "$src" -j"$jobs_count" >&2 || die "$slot: build of $short failed" + + mkdir -p "$dst" + cp "$src/profanity2.x64" "$src/keccak.cl" "$src/profanity.cl" "$dst/" + fi + + printf '%s' "$ref" > "$WORKDIR/$slot.ref" + printf '%s' "$short" > "$WORKDIR/$slot.sha" +} + +mkdir -p "$WORKDIR" + +build_slot a "$REF_A" "$SHA_A" "$SHORT_A" +build_slot b "$REF_B" "$SHA_B" "$SHORT_B" + +# Whether the two revisions time a round the same way. If they do not, the +# speeds they print come from different clocks and the runner warns about it. +if git -C "$SRC_REPO" diff --quiet "$SHA_A" "$SHA_B" -- SpeedSample.cpp; then + printf 'same\n' > "$WORKDIR/timer.state" +else + printf 'differs\n' > "$WORKDIR/timer.state" + log "warning: SpeedSample.cpp differs between the two revisions" +fi + +log "ready, run the comparison with:" +log " BENCH_ROOT=$WORKDIR $script_dir/run-benchmark.sh" + +printf '%s\n' "$WORKDIR" diff --git a/bench/run-benchmark.sh b/bench/run-benchmark.sh index 1168ff1..8d05fa9 100755 --- a/bench/run-benchmark.sh +++ b/bench/run-benchmark.sh @@ -1,6 +1,7 @@ #!/usr/bin/env bash # -# Compares the speed of the two profanity2 revisions baked into this image. +# Compares the speed of two profanity2 revisions laid out side by side, either +# baked into this image or prepared on the host by bench/prepare-native.sh. # # The two builds never run at the same time and never share a kernel cache. # They take turns - A, B, A, B - so that clock drift or thermal throttling on @@ -22,14 +23,18 @@ # Every option can also be given as an environment variable (BENCH_MODE, # BENCH_SECONDS, BENCH_WARMUP, BENCH_REPEATS, BENCH_EXTRA_ARGS, # BENCH_EXACT_MASK, BENCH_PUBLIC_KEY), which is the only way to configure the -# run on platforms that replace the image entrypoint. +# run on platforms that replace the image entrypoint. BENCH_ROOT points at the +# directory holding the two revisions [default = /opt/bench]. # # An argument that does not start with a dash is executed instead of the # benchmark, e.g. `clinfo` or `bash`. set -euo pipefail -readonly ROOT=/opt/bench +# The benchmark image lays the two revisions out in /opt/bench. A run on the +# host machine has no such directory and passes its own in BENCH_ROOT, which is +# what bench/prepare-native.sh prints. +readonly ROOT="${BENCH_ROOT:-/opt/bench}" # Generator point of secp256k1, x and y concatenated without the 04 prefix. It # is a valid public key, so profanity2 accepts it, and its private key is the @@ -80,7 +85,7 @@ usage() { cat <<'EOF' usage: bench [OPTIONS] -Runs the two profanity2 revisions baked into this image one after the other, +Runs the two profanity2 revisions found in BENCH_ROOT one after the other, alternating A B A B, and reports how their speeds compare. --mode leading|exact what to measure [default = leading] @@ -100,6 +105,10 @@ run on platforms that replace the image entrypoint. A plain PUBLIC_KEY is used too, but only when it holds 128 hexadecimal characters, because rental platforms hand out that name for their own SSH key. +BENCH_ROOT is the directory holding the two revisions [default = /opt/bench]. +Outside the image, bench/prepare-native.sh builds that directory and prints +its path. + An argument that does not start with a dash is executed instead of the benchmark, e.g. `clinfo` or `bash`. EOF @@ -130,6 +139,12 @@ count_opencl_platforms() { count="$(clinfo 2>/dev/null | awk '/^Number of platforms/ { print $NF; exit }' || true)" ;; esac + elif [ "$(uname -s)" = Darwin ]; then + # macOS ships OpenCL.framework and no clinfo, so there is no ICD + # loader to interrogate and nothing to count. Whether a usable + # device exists is left to profanity2's own enumeration, which + # prints the device list and is caught by wait_for_first_sample. + count=1 fi case "$count" in @@ -287,10 +302,16 @@ main() { platforms=1 fi if [ "$platforms" -eq 0 ]; then - log "no OpenCL platform found inside the container" - log " installed ICDs: $(echo /etc/OpenCL/vendors/*.icd)" - log " start the container with GPU access (docker run --gpus all ...)" - log " or run this image with the argument \"clinfo\" to see what it finds" + if [ -n "${BENCH_ROOT:-}" ]; then + log "no OpenCL platform found on this machine" + log " install clinfo to see what the ICD loader finds, or set" + log " BENCH_SKIP_GPU_CHECK=1 to run anyway" + else + log "no OpenCL platform found inside the container" + log " installed ICDs: $(echo /etc/OpenCL/vendors/*.icd)" + log " start the container with GPU access (docker run --gpus all ...)" + log " or run this image with the argument \"clinfo\" to see what it finds" + fi exit 1 fi @@ -309,21 +330,28 @@ main() { fi echo - local -A speeds=([a]="" [b]="") - local repeat slot speed + # Two plain variables rather than an associative array: macOS ships bash + # 3.2, where `local -A` and ${slot^^} do not exist. + local speeds_a="" speeds_b="" + local repeat slot slot_name speed for repeat in $(seq "$REPEATS"); do for slot in a b; do - log "run $repeat/$REPEATS, revision ${slot^^} ($(label "$slot"))" + slot_name="$(printf '%s' "$slot" | tr 'a-z' 'A-Z')" + log "run $repeat/$REPEATS, revision $slot_name ($(label "$slot"))" if ! speed="$(run_once "$slot")"; then - die "revision ${slot^^} failed to produce a measurement" + die "revision $slot_name failed to produce a measurement" + fi + log "run $repeat/$REPEATS, revision $slot_name: $(format_speed "$speed")" + if [ "$slot" = a ]; then + speeds_a="$speeds_a $speed" + else + speeds_b="$speeds_b $speed" fi - log "run $repeat/$REPEATS, revision ${slot^^}: $(format_speed "$speed")" - speeds[$slot]="${speeds[$slot]} $speed" done done - report "${speeds[a]}" "${speeds[b]}" + report "$speeds_a" "$speeds_b" } spread_pct() { From f70e8e7e7ca5e04dff3db5d51eb4809ae63fce30 Mon Sep 17 00:00:00 2001 From: Gleb Alekseev Date: Sat, 8 Aug 2026 19:09:12 +0200 Subject: [PATCH 2/2] Record an M4 Max comparison of 9011bcd and pull request 57 The two revisions come out even on an Apple GPU - medians 252.6 against 252.0 MH/s - so the roughly 9% that pull request 57 gains on an RTX 3060 does not carry over to Metal's OpenCL. The runner flags the run as noise, and on the reported spread it is right: a cold first run for A and a high third run for B stretch it to 44% and 58%. The medians are robust to both, and the last three pairs, measured once the machine had settled, sit between 236 and 253 MH/s for either revision. That resolves a regression of about 5% or larger and nothing finer, which is enough for what the run was for. Co-authored-by: Cursor --- bench-logs/M4Max.mac.log | 43 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 bench-logs/M4Max.mac.log diff --git a/bench-logs/M4Max.mac.log b/bench-logs/M4Max.mac.log new file mode 100644 index 0000000..b304075 --- /dev/null +++ b/bench-logs/M4Max.mac.log @@ -0,0 +1,43 @@ +profanity2 benchmark + A: 9011bcd (9011bcd) + B: pr57-head (aad8e13) + workload: --leading 0 -i 255 -I 16384 -w 64 + window: 6 x 75s per revision, first 60s of each run dropped + order: A B A B A B A B A B A B + key: the secp256k1 generator + a key everybody knows - never use a result from this run +bench: run 1/6, revision A (9011bcd (9011bcd)) +bench: run 1/6, revision A: 339.9 MH/s +bench: run 1/6, revision B (pr57-head (aad8e13)) +bench: run 1/6, revision B: 221.3 MH/s +bench: run 2/6, revision A (9011bcd (9011bcd)) +bench: run 2/6, revision A: 252.2 MH/s +bench: run 2/6, revision B (pr57-head (aad8e13)) +bench: run 2/6, revision B: 272.5 MH/s +bench: run 3/6, revision A (9011bcd (9011bcd)) +bench: run 3/6, revision A: 264.2 MH/s +bench: run 3/6, revision B (pr57-head (aad8e13)) +bench: run 3/6, revision B: 349.3 MH/s +bench: run 4/6, revision A (9011bcd (9011bcd)) +bench: run 4/6, revision A: 253.0 MH/s +bench: run 4/6, revision B (pr57-head (aad8e13)) +bench: run 4/6, revision B: 250.4 MH/s +bench: run 5/6, revision A (9011bcd (9011bcd)) +bench: run 5/6, revision A: 241.3 MH/s +bench: run 5/6, revision B (pr57-head (aad8e13)) +bench: run 5/6, revision B: 240.6 MH/s +bench: run 6/6, revision A (9011bcd (9011bcd)) +bench: run 6/6, revision A: 236.5 MH/s +bench: run 6/6, revision B (pr57-head (aad8e13)) +bench: run 6/6, revision B: 253.5 MH/s +=================== BENCHMARK RESULT =================== +A 9011bcd (9011bcd) median 252.6 MH/s spread 43.7% + runs: 339.9 MH/s 252.2 MH/s 264.2 MH/s 253.0 MH/s 241.3 MH/s 236.5 MH/s +B pr57-head (aad8e13) median 252.0 MH/s spread 57.8% + runs: 221.3 MH/s 272.5 MH/s 349.3 MH/s 250.4 MH/s 240.6 MH/s 253.5 MH/s +B vs A: -0.3% +warning: the difference is no larger than the spread between repeats of + the same revision (57.8%), so it cannot be told apart from + noise - use a longer --seconds, more --repeats or another host +======================================================== +BENCH_RESULT mode=leading a_hs=252590750 b_hs=251952750 delta_pct=-0.3