Skip to content

embeddinggemma.c v0.2.4

Choose a tag to compare

View all tags
@ehartford ehartford released this 20 Jul 20:39
v0.2.4
4749337

Highlights

  • Added a production ROCm backend with packed-Q4 kernels, native MFMA routes,
    fused short-request paths, batched hipBLAS projections and attention, and
    dynamic packed batching.
  • The ROCm release executable is portable across CDNA1-CDNA4 and contains code
    objects for gfx908, gfx90a, gfx942, and gfx950.
  • The installer now detects ROCm on Linux and falls back to CPU when an
    accelerator runtime is unavailable.
  • Added CONTRIBUTING.md, a per-platform RELEASE.md runbook, model-free CI,
    backend clean/staging targets, complete-asset verification, and deterministic
    checksums.
  • Darwin binaries now explicitly target macOS 14.0 instead of inheriting the
    build host's current SDK minimum.

Performance

On the tested AMD Instinct MI300X, the retained ROCm engine measured about
1,072 req/s at one token, 8,890 input tok/s at 32 tokens, and 206,011 input
tok/s at 2,048 tokens. A packed batch of 32 independent 32-token requests
reached 5,161 req/s and 165,164 input tok/s.

These are hardware-specific warmed measurements. Full commands and A/B results
are recorded in perf/optimization_status.md.

Release Assets

All binary assets are raw stripped executables:

  • Darwin ARM64 CPU, targeting macOS 14.0+
  • Darwin ARM64 Metal with its metallib embedded, targeting macOS 14.0+
  • Linux x86_64 CPU
  • Linux x86_64 CUDA with 19 native cubins from SM50 through SM121 and
    compute_121 PTX
  • Linux x86_64 ROCm with CDNA gfx908, gfx90a, gfx942, and gfx950
  • Linux x86_64 XPU SYCL with PVC and BMG device images
  • SHA256SUMS covering exactly those six executables

CUDA requires the CUDA 12 runtime and cuBLAS. ROCm requires HIP, hipBLAS, and
the AMDGPU runtime. XPU requires Intel oneAPI 2026.1 SYCL/oneMKL and Level Zero.
The installer automatically falls back to the Linux CPU asset when an
auto-selected accelerator has missing shared-library dependencies.

Validation

CPU, Metal, CUDA, ROCm, and XPU passed llama.cpp golden parity, backend drift,
packed-batch, and HTTP Matryoshka dimension tests. Minimum golden cosine was
above 0.99986 on every backend. The published artifacts were rebuilt from
commit 4749337ab9a195f3e023a5a75e75dd10d571e476, stripped, format-checked,
and checksum-verified. Both Darwin executables were ad-hoc signed and verified.

This release contains no model weights. Users acquire and use the model under
the model provider's terms.

Assets 9
Loading