Skip to content

feat: Add dedicated GitHub Actions Performance Benchmarking workflow#30

Merged
solderzzc merged 1 commit intomainfrom
feature/ci-benchmarks
Apr 12, 2026
Merged

feat: Add dedicated GitHub Actions Performance Benchmarking workflow#30
solderzzc merged 1 commit intomainfrom
feature/ci-benchmarks

Conversation

@solderzzc
Copy link
Copy Markdown
Member

Overview

This PR introduces a dedicated, standalone GitHub Actions workflow (.github/workflows/benchmark.yml) designed to automatically benchmark Apple MLX generation speeds directly on the CI infrastructure (macos-15 runner).

Since Cloud macOS runners typically possess exactly 7 GB of RAM, blindly running the standard benchmark script matrix risks immediate Out-Of-Memory (OOM) SIGKILL boundaries. This safe workflow explicitly forces constraints and exposes a configurable matrix to test safe parameter bounds directly from the UI.

Features

  • UI Triggered (workflow_dispatch): Fully configurable directly through the GitHub Actions web UI. Does not run implicitly on pushes.
  • Configurable Models: Supports swapping the model_id input immediately prior to run. Defaults to the un-gated mlx-community/gemma-4-e4b-it-4bit which safely drops into memory constraints.
  • Context Boundaries: Tests generation strictly against safe, small context lengths ("512,1024,4096") by default.
  • SSD streaming isolation: Toggles the --ssd-only parameter to explicitly bypass physical limits and test extreme combinations.

Outputs

  • Downloads and manages dependencies (mlx, psutil, requests) inside the GH Action.
  • Bundles the final generated github-action-benchmark.md table securely as an exported execution artifact on each run.

@solderzzc solderzzc merged commit 00ce868 into main Apr 12, 2026
3 checks passed
@solderzzc solderzzc deleted the feature/ci-benchmarks branch April 12, 2026 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant