feat(benchmarks): macOS benchmark suite + fresh-install and review fixes#53
Merged
Conversation
Add macOS/benchmarks/ with synthetic CPU/memory/storage benchmarks, a thermal stress/throttle test, standardized benchmark wrappers (Geekbench 6, Cinebench, Blender), a local-LLM benchmark across MLX and llama.cpp, and an oMLX continuous-batching concurrency benchmark. A suite-aware compare.sh diffs two machines' results. Wire new optional macOS tools into packages.json (fio, hyperfine, stress-ng, geekbench, cinebench, blender-benchmark) and install the oMLX menu-bar app from its GitHub .dmg via macOS/install-omlx-app.sh. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wrap the Geekbench AI `banff` CLI (--ai) in standardized.sh alongside Geekbench 6/Cinebench/Blender, recording the result URL (free CLI uploads to the Geekbench AI Browser). Add the geekbench-ai cask to packages.json and a Geekbench AI score row to compare.sh. Document that oMLX's built-in /admin benchmark (performance + intelligence) is dashboard-only; omlx-bench.sh is the scriptable counterpart for the performance (PP/TG tokens/sec) side. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add "benchmarking" to the controlled tag vocabulary in scripts/validate-packages.sh and retag the benchmark/stress-test tools (fio, hyperfine, stress-ng, geekbench, geekbench-ai, cinebench, blender-benchmark) from system-monitoring to benchmarking. The interactive selector picks it up automatically (vocabulary is derived from packages.json), so `--tags benchmarking` installs just these. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Prime sudo once with a background keepalive so cask/pkg installers reuse the cached credential instead of prompting ~6 times. - Resolve the oMLX DMG by macOS generation (macos15-sequoia vs macos26-27) via sw_vers instead of blindly taking head -1. - Stop auto-running `codeburn menubar` (buggy upstream); print it as a manual post-install step. codeburn itself still installs. - Add [i/N] progress counters to brew, cask, and pipx install tiers. - Install cinebench from Maxon's .dmg (custom) instead of the brew cask, whose pinned checksum goes stale against Maxon's rolling build (SHA256 mismatch). Update the two stale cask references in the docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Correctness: - standardized.sh: Blender guard `[[ ... && ! $CPU_ONLY ]]` was always false (string negation) so Blender never ran; use `&& ! $CPU_ONLY`. - benchmark.sh: fio scratch trap RETURN → EXIT (RETURN never fires at script scope, leaking a multi-GB dir); guard empty openssl parse → 0. - stress-test.sh: floor SAMPLE_COUNT to ≥1 (short DURATION divided by zero); die on empty/zero baseline; guard empty per-sample throughput. - standardized.sh: capture ts_file once so OUTFILE/RAWDIR can't straddle a second boundary and mismatch. - omlx-bench.sh: now_s() falls back to `date +%s` when perl is absent so jq -n interpolation never gets an empty operand; print "n/a" instead of 0.0 when there is no single-stream sample. - llm-bench.sh: `.avg_ts | tonumber?` so a null row can't abort the run. - platforms/macos.sh: gate mac_custom_install_tier on handled_by_setup, matching the documented custom-manager contract. Altitude / cleanup: - install-omlx-app.sh: pick the DMG whose macosNN is the highest ≤ this Mac's major version (was hardcoded 26/27, fell through on 28+). - lib-dmg-install.sh: extract the shared DMG download/mount/copy helper; both installers source it. Trap bakes paths in at register time so the EXIT cleanup works under set -u after the function returns. - lib.sh: drop unused check_dep_optional. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ulises-c
changed the title
…ndings Full-branch max-effort review (10 finder angles + per-finding verification + gap sweep) of the macOS benchmark suite. P0: broken measurement paths (LibreSSL -seconds, llama-bench --hf-repo, GPU/mem-bw/powermetrics parses, Cinebench find depth), data corruption (compare.sh crash, omlx silent request failures, Cinebench single-core bleed), setup aborts, and a public-repo hostname leak. P1/P2: moderate and latent correctness. P3: dedupe cleanups. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- lib.sh: resolve_openssl picks brew openssl@3 / real OpenSSL — stock LibreSSL lacks `speed -seconds`; all speed substitutions now || true - benchmark.sh: drop the GPU section (llm-bench.sh owns tokens/s with a pinned model); anchor the stress-ng stream parse on the metrc row - llm-bench.sh: llama-bench takes no --hf-repo — resolve the quant's .gguf via the HF API, cache it, pass -m; keep stderr for diagnostics - standardized.sh: -maxdepth 4 so the Cinebench binary is findable; separate raw files per Cinebench pass (no multi→single score bleed); run single-core under --cpu-only (it is a CPU test); download the Blender runtime + scenes before benchmarking - stress-test.sh: baseline measured under load after clocks settle (idle baseline read contention as throttling); Apple Silicon powermetrics patterns (HW active frequency / CPU Power, mW→W) - compare.sh: null pct renders n/a instead of crashing the table (@TSV empty-field collapse under IFS tab); add the stress suite; header comment lists all five suites - omlx-bench.sh: fire_one reports failures; per-level failed_requests recorded; failed levels excluded from peak/speedup - platforms/macos.sh: custom tier collects failures like the brew tiers instead of aborting the remaining setup run under set -e - lib/verify.sh: macOS custom probe also checks /Applications .app bundles so cinebench/omlx verify after install - README: placeholder hostnames + correct results path in the compare example; docs updated for openssl@3, GGUF cache, loaded baseline Validated: shellcheck + bash -n clean, dryrun-smoke passes all four platforms, compare.sh crash/stress paths exercised with fixture files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- llm-bench.sh: guard the MLX PP/TG/MEM parse pipelines with || true so stats-format drift records null instead of dying under pipefail - stress-test.sh / omlx-bench.sh: INT/TERM traps now exit (cleanup runs once via the EXIT trap) — a mid-run kill previously stopped the load but kept sampling an idle machine and wrote a bogus throttled:false result; verified with a PID-directed kill test - omlx-bench.sh: pass --port to `omlx serve` so an OMLX_PORT override polls the port the server actually binds - lib.sh: ensure_results_dir chowns results/ to SUDO_USER so a sudo stress-test run doesn't make every later non-sudo suite die at its final result write - compare.sh: metrics measured on only one machine render Win=n/a and stay out of the A/B win tally; drop the Geekbench AI row (the free CLI never yields a numeric score, only a browser URL) - benchmark.sh: scaling_factor via --argjson — a real number (or null) in the result JSON instead of a string Validated: shellcheck + bash -n clean; compare.sh fixture run shows the one-sided row as n/a and excluded from the tally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- platforms/macos.sh: call custom_reminders_section like linux_main — custom entries the engine does not run are now surfaced as manual reminders (mongodb-community appears on macOS for the first time) instead of silently dropped; gate the pipx [i/N] progress line on DRY_RUN; print a [dry-run] line for the sudo priming step; progress and FAIL lines for casks show the --adopt flag actually executed; keepalive refresh gets || true (inherited set -e) and stdout is redirected so piped runs see EOF at exit - scripts/validate-packages.sh: handled_by_setup must be boolean if present (the engine tests == true, so a string "true" silently downgraded to a reminder); custom entries must resolve a string install_command for every platform they target - macOS/lib-dmg-install.sh: trust the mount point hdiutil reports — it reuses an existing mount and ignores -mountpoint when the image is already attached (e.g. opened in Finder); cleanup re-registers with the mount actually used - standardized.sh: free-tier Geekbench fallback greps the captured output for the browser URL before paying a second full CPU run, and appends instead of truncating the raw log - omlx-bench.sh: with an explicit OMLX_MODEL a responding API counts as ready — lazy-loading servers list no models until first request Validated: validate-packages passes the real packages.json and a negative fixture catches all three new bad shapes; shellcheck + bash -n clean; dryrun-smoke passes (macOS 87→88 actions: the new [dry-run] sudo line); the macOS dry run shows the mongodb-community reminder. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- benchmarks/lib.sh: bench_init owns the suite prologue (results dir, SYSINFO, HOSTNAME_SHORT/STAMP/OUTFILE, system banner) previously copy-pasted across all five suites; SUITE_VERSION replaces five "1.0.0" literals; openssl_sha256_kbs replaces the four copies of the openssl-speed run+parse in benchmark.sh/stress-test.sh - platforms/macos.sh: mac_install_list is the single [i/N] install loop behind the brew/cask/pipx tiers (was three near-identical copies); as a side effect pipx failures now collect into the run summary instead of aborting setup under set -e - codeburn menubar reminder is data-driven: new codeburn-menubar custom entry (priority none, handled_by_setup false) rendered by custom_reminders_section; the hardcoded name-check block in platform_main is gone - lib-dmg-install.sh: install_app_from_dmg <AppName> <url> now owns the already-installed guard, curl/hdiutil dep checks, and the success line; both installers shrink to URL resolution (omlx keeps an early app_installed guard to skip the rate-limited GitHub API call on re-runs). info/die stay local — sourcing benchmarks/lib.sh would couple standalone installers to the suite's internals - install-cinebench.sh: DMG_URL overridable via CINEBENCH_DMG_URL for when Maxon ships the next major version Validated: validate-packages passes (106 entries), shellcheck + bash -n clean, dryrun-smoke passes all four platforms (88 macOS actions unchanged), dry run shows the codeburn-menubar reminder and 53 [i/N] lines through the shared loop; lib/core.sh untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ulises-c
marked this pull request as ready for review
July 1, 2026 22:58
This was referenced Jul 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a macOS benchmarking suite and integrates its tooling into the installer, then hardens it against a round of fresh-install feedback and a max-effort code review.
Benchmark suite (
macOS/benchmarks/):benchmark.sh(CPU/memory/storage/GPU),stress-test.sh(sustained load + throttle detection),llm-bench.sh(MLX + llama.cpp),omlx-bench.sh(concurrency sweep),standardized.sh(Geekbench 6 / Geekbench AI / Cinebench / Blender wrappers), pluscompare.sh,collect-sysinfo.sh, and sharedlib.sh. Newbenchmarkingtag category inpackages.json.Fresh-install fixes (
e60aa79):sudoonce with a background keepalive so cask/pkg installers stop prompting ~6 times.codeburn menubar; print it as a manual step.[i/N]progress counters to the brew/cask/pipx install tiers..dmg(custom) instead of the brew cask, whose pinned checksum goes stale against Maxon's rolling build.Review-finding fixes (
3cba935):standardized.sh: Blender guard[[ … && ! $CPU_ONLY ]]was always false (string negation) so Blender never ran — corrected to&& ! $CPU_ONLY.benchmark.sh: fio scratchtrap … RETURN→EXIT(RETURN never fires at script scope, leaking a multi-GB dir); guard emptyopensslparse → 0.stress-test.sh: floorSAMPLE_COUNTto ≥1 (short duration divided by zero); die on empty/zero baseline; guard empty per-sample throughput.standardized.sh: capturets_fileonce soOUTFILE/RAWDIRcan't straddle a second boundary.omlx-bench.sh:now_s()falls back todate +%swhen perl is absent; printn/ainstead of0.0when there is no single-stream sample.llm-bench.sh:.avg_ts | tonumber?so a null row can't abort the run underset -e.platforms/macos.sh: gatemac_custom_install_tieronhandled_by_setup, matching the documented custom-manager contract.install-omlx-app.sh: pick the DMG whosemacosNNis the highest ≤ this Mac's major version (was hardcoded 26/27, fell through on 28+).macOS/lib-dmg-install.sh: shared DMG download/mount/copy helper sourced by both installers; trap bakes paths in at register time so EXIT cleanup works underset -u.lib.sh: drop unusedcheck_dep_optional.Test plan
shellcheck --severity=warningclean on all shell scripts.scripts/validate-packages.shPASSED (105 entries).scripts/dryrun-smoke.shPASSED —setup.sh --dry-runclean on macos/ubuntu/arch/server.set -esafety of the arithmetic guards, Blender truth table,tonumber?null-dropping, and the DMG-helper EXIT-trap underset -u.Caveat: the benchmark scripts themselves have not been run live on a Mac (no
hdiutil/openssl speed/powermetricsin CI). The LibreSSLopenssl speedcolumn question is now safely guarded (empty → 0/die) but not yet confirmed against a real run.🤖 Generated with Claude Code