embeddinggemma.c v0.3.1

What's new

• Adds an OpenAI-compatible POST /v1/embeddings endpoint with exact token usage, OpenAI error envelopes, Matryoshka dimensions, float/base64 encodings, and cache isolation from /api/embed.
• Adds a persistent exact-embedding cache via --persistent-cache-path for repeat, re-index, and evaluation workloads.
• Enables fused Metal up/gate/GELU by default.
• Adds opt-in high-token CUDA W4A8 projection and int8 attention paths (EI_CUDA_W4A8_GEMM, EI_CUDA_INT8_ATTN).
• Adds opt-in XPU attention-output half fusion (EI_XPU_FUSE_ATTN_OUTPUT_HALF).
• Corrects llama.cpp benchmark ordering/thermal bias with both-order measurement support and documents the measured optimization ceiling.

Assets

This release contains raw executables for Darwin ARM64 CPU and Metal, plus Linux x86_64 CPU, CUDA, ROCm, and XPU. SHA256SUMS covers exactly those six binaries. No model weights are included.

Validation

Built from commit 19a5da8e8499 and tested on Apple M5 Max, NVIDIA RTX 3090 (GPU 1), AMD MI300X, and Intel Arc Pro B60. All backend golden, route-parity, packed-batch, and native/OpenAI HTTP tests passed. CUDA includes native cubins for SM 5.0 through SM 12.1 plus SM 12.1 PTX; ROCm includes gfx908, gfx90a, gfx942, and gfx950; XPU includes PVC and BMG device images.

v0.3.0 — Faster than llama.cpp on every backend

embeddinggemma.c v0.3.0

Faster than llama.cpp across the board, on every backend.

This release lands a batch-scheduler overhaul that lifts throughput on CPU,
Metal, CUDA, ROCm, and XPU, plus new Metal 4 kernels and a serving-path
rewrite. Measured over HTTP against llama.cpp b8981 on an Apple M5 Max —
identical GGUF, exact token counts, caches disabled, fresh servers per cell —
embeddinggemma is faster in all 54 shapes, geometric mean 1.25x,
range 1.01x–2.01x.

Highlights

• Demand-aware wave batch collection. The dynamic batcher now tracks peak
  in-flight demand and dispatches on wave completion instead of a fixed timer,
  fixing an alternating-singleton pathology that suppressed batching at
  concurrency 2 on every backend. Measured lift at c2: CPU +18%, CUDA +78%,
  ROCm +64%, XPU +55%. Wins of +15–35% across higher concurrency.
• Metal 4 tensor projections and flash attention (Apple GPUs reporting
  MTLGPUFamilyMetal4), embedded alongside the Metal 3.1 library with runtime
  family selection so older Apple Silicon keeps working.
• Serving-path rewrite: Ryu shortest-round-trip float formatting (~3x
  faster responses), single-writev responses with TCP_NODELAY,
  SO_RCVTIMEO keep-alive, and split Metal command encoding for CPU/GPU
  overlap.
• Correctness: a single-flight fix for concurrent inline/backend execution;
  validated across all backends (min cross-backend cosine ≥ 0.999, 1,000,012
  exact float round-trips).

Fixes

• Linux build: include <sys/time.h> for struct timeval in the keep-alive
  path (every Linux target failed to build under -Werror otherwise).
• README examples now use the correct EmbeddingGemma prompt prefixes
  (task: search result | query: / title: none | text:).

Assets

Raw executables, one per platform/backend. Verify with checksums.txt.

platform	backend	asset
macOS ARM64	CPU	embeddinggemma-darwin-arm64-cpu
macOS ARM64	Metal	embeddinggemma-darwin-arm64-metal
Linux x86_64	CPU	embeddinggemma-linux-x86_64-cpu
Linux x86_64	CUDA	embeddinggemma-linux-x86_64-cuda
Linux x86_64	ROCm	embeddinggemma-linux-x86_64-rocm
Linux x86_64	XPU SYCL	embeddinggemma-linux-x86_64-xpu

embeddinggemma.c v0.2.5

Canonical quixiembed port

quixiembed now listens on port 42666 by default on every backend. The port
was selected with the macOS jot random-number utility from the non-privileged,
non-ephemeral range and verified unused on the selection host.

This avoids colliding with Ollama's default port 11434. Deployments that need
a different endpoint can continue to pass --port PORT.

The compiled default is exposed by --help, documented in the serving
contract, and enforced by the model-free CI suite so backend binaries cannot
silently drift to different ports.

Release assets

All binary assets are raw stripped executables rebuilt from commit
712956cc56a9be2cfe236a38b4800d8e43e65cac:

• Darwin ARM64 CPU, targeting macOS 14.0+
• Darwin ARM64 Metal with its metallib embedded, targeting macOS 14.0+
• Linux x86_64 CPU
• Linux x86_64 CUDA with 19 native cubins from SM50 through SM121 and
  compute_121 PTX
• Linux x86_64 ROCm with CDNA gfx908, gfx90a, gfx942, and gfx950
• Linux x86_64 XPU SYCL with PVC and BMG device images
• SHA256SUMS covering exactly those six executables

Every staged executable reports default listen: 0.0.0.0:42666 and passed its
backend HTTP Matryoshka dimension test. Both Darwin binaries were ad-hoc signed
and verified. Accelerator dependency checks and executable-format validation
also passed.

This release contains no model weights. Users acquire and use the model under
the model provider's terms.

embeddinggemma.c v0.2.4

Highlights

• Added a production ROCm backend with packed-Q4 kernels, native MFMA routes,
  fused short-request paths, batched hipBLAS projections and attention, and
  dynamic packed batching.
• The ROCm release executable is portable across CDNA1-CDNA4 and contains code
  objects for gfx908, gfx90a, gfx942, and gfx950.
• The installer now detects ROCm on Linux and falls back to CPU when an
  accelerator runtime is unavailable.
• Added CONTRIBUTING.md, a per-platform RELEASE.md runbook, model-free CI,
  backend clean/staging targets, complete-asset verification, and deterministic
  checksums.
• Darwin binaries now explicitly target macOS 14.0 instead of inheriting the
  build host's current SDK minimum.

Performance

On the tested AMD Instinct MI300X, the retained ROCm engine measured about
1,072 req/s at one token, 8,890 input tok/s at 32 tokens, and 206,011 input
tok/s at 2,048 tokens. A packed batch of 32 independent 32-token requests
reached 5,161 req/s and 165,164 input tok/s.

These are hardware-specific warmed measurements. Full commands and A/B results
are recorded in perf/optimization_status.md.

Release Assets

All binary assets are raw stripped executables:

• Darwin ARM64 CPU, targeting macOS 14.0+
• Darwin ARM64 Metal with its metallib embedded, targeting macOS 14.0+
• Linux x86_64 CPU
• Linux x86_64 CUDA with 19 native cubins from SM50 through SM121 and
  compute_121 PTX
• Linux x86_64 ROCm with CDNA gfx908, gfx90a, gfx942, and gfx950
• Linux x86_64 XPU SYCL with PVC and BMG device images
• SHA256SUMS covering exactly those six executables

CUDA requires the CUDA 12 runtime and cuBLAS. ROCm requires HIP, hipBLAS, and
the AMDGPU runtime. XPU requires Intel oneAPI 2026.1 SYCL/oneMKL and Level Zero.
The installer automatically falls back to the Linux CPU asset when an
auto-selected accelerator has missing shared-library dependencies.

Validation

CPU, Metal, CUDA, ROCm, and XPU passed llama.cpp golden parity, backend drift,
packed-batch, and HTTP Matryoshka dimension tests. Minimum golden cosine was
above 0.99986 on every backend. The published artifacts were rebuilt from
commit 4749337ab9a195f3e023a5a75e75dd10d571e476, stripped, format-checked,
and checksum-verified. Both Darwin executables were ad-hoc signed and verified.

This release contains no model weights. Users acquire and use the model under
the model provider's terms.

embeddinggemma.c v0.2.3

v0.2.3 adds the optimized Intel XPU SYCL backend, completes the cross-platform
benchmark documentation, and ships the first installer-aware XPU release.
Release assets are raw executables; GitHub provides source archives separately.
Model weights are not included.

Binaries

• embeddinggemma-darwin-arm64-cpu: Apple Silicon CPU server.
• embeddinggemma-darwin-arm64-metal: Apple Silicon Metal server with the
  compiled Metal library embedded in the executable.
• embeddinggemma-linux-x86_64-cpu: Linux x86-64 CPU server.
• embeddinggemma-linux-x86_64-cuda: Linux x86-64 CUDA server with native cubins
  for every architecture supported by CUDA 12.9, plus newest-target PTX.
• embeddinggemma-linux-x86_64-xpu: Linux x86-64 oneAPI XPU SYCL server with
  PVC/BMG device images and runtime-gated Xe2 Flash, W4A16, command-graph, and
  register-cached RMS optimizations.
• SHA256SUMS: SHA-256 checksums for all five executables.

The installer saves the selected release binary as ~/.local/bin/quixiembed:

curl -fsSL https://raw.githubusercontent.com/QuixiAI/embeddinggemma.c/main/install.sh | sh

Runtime Requirements

The Darwin binaries target macOS 14.0 or newer and are ad-hoc signed. The Metal
binary has no metallib sidecar. Linux model acquisition invokes an available
curl or wget command and has no linked libcurl dependency.

The CUDA binary requires CUDA 12 runtime libraries and an NVIDIA driver. It
contains native SM50, SM52, SM53, SM60, SM61, SM62, SM70, SM72, SM75, SM80,
SM86, SM87, SM89, SM90, SM100, SM101, SM103, SM120, and SM121 code plus
compute_121 PTX.

The XPU binary requires the Intel oneAPI 2026.1 SYCL/oneMKL runtime and a Level
Zero driver. It was validated on Intel Arc Pro B60; BMG-specific routes are
enabled only after an exact runtime architecture check.

Validation

• CPU, Metal, CUDA, and XPU pass all 10 llama.cpp golden embeddings.
• Packed-batch and backend-specific kernel parity suites pass.
• The stripped release executables pass HTTP Matryoshka dimensions 768, 512,
  256, and 128 on their native validation hosts.
• Metal loads its embedded kernels with no sidecar.
• CUDA contains all 19 advertised cubins and its PTX fallback.
• XPU was rebuilt from the pinned vLLM XPU kernel and SYCL-TLA commits.

The README now records comparable warmed engine throughput for Apple M5 Max
CPU/Metal, RTX 3090 CUDA, and Arc Pro B60 XPU, together with concurrent and
packed short-request throughput.

embeddinggemma.c v0.2.2

This patch release replaces the SM86-only CUDA artifact with a portable CUDA
fat binary. Release assets are raw executables, not tarballs. GitHub provides
the source .tar.gz and .zip archives separately.

Binaries

• embeddinggemma-darwin-arm64-cpu: Apple Silicon CPU server.
• embeddinggemma-darwin-arm64-metal: Apple Silicon Metal server with all
  Metal kernels embedded in the executable.
• embeddinggemma-linux-x86_64-cpu: Linux x86-64 CPU server.
• embeddinggemma-linux-x86_64-cuda: Linux x86-64 CUDA server containing
  native code for every architecture supported by CUDA 12.9, plus PTX for its
  newest target.
• SHA256SUMS: SHA-256 checksums for the four executables.

Downloaded executables may need chmod +x <filename> before use. Model weights
are not included.

The Darwin binaries target macOS 14.0 or newer and are ad-hoc signed. The Linux
binaries were built on Ubuntu 24.04 and dynamically use system libcurl. The
CUDA binary additionally requires CUDA 12 libraries and an NVIDIA driver; it
was built with CUDA 12.9 and validated on an RTX 3090.

CUDA Portability

The CUDA executable contains native cubins for SM50, SM52, SM53, SM60, SM61,
SM62, SM70, SM72, SM75, SM80, SM86, SM87, SM89, SM90, SM100, SM101, SM103,
SM120, and SM121, plus compute_121 PTX for forward compatibility.

Packed Q4/Q8 projection uses dp4a on compute capability 6.1 and newer and a
scalar integer fallback on older targets. The optional native packed-Q4 MMA
path now reports a clear error below compute capability 8.0 instead of exposing
Ampere-only instructions to older GPUs. Architectures other than SM86 were
compile-validated but were not hardware-tested for this release.

Validation

• CPU, Metal, and CUDA pass all 10 llama.cpp golden embeddings.
• CPU, Metal, and CUDA pass packed-batch parity.
• HTTP Matryoshka dimensions 768, 512, 256, and 128 pass using every stripped
  release executable.
• Metal passes the full backend parity suite without a metallib sidecar.
• CUDA attention, QKV epilogue, Q4 MMA, and Q8 latency route parity pass.

embeddinggemma.c v0.2.1

Superseded by v0.2.2, which replaces the SM86-only CUDA asset with a portable CUDA fat binary.

This patch release corrects binary packaging and makes the Metal server a
single self-contained executable. Release assets are raw executables, not
tarballs. GitHub provides the source .tar.gz and .zip archives separately.

Binaries

• embeddinggemma-darwin-arm64-cpu: Apple Silicon CPU server.
• embeddinggemma-darwin-arm64-metal: Apple Silicon Metal server with all
  Metal kernels embedded in the executable.
• embeddinggemma-linux-x86_64-cpu: Linux x86-64 CPU server.
• embeddinggemma-linux-x86_64-cuda-sm86: Linux x86-64 CUDA server compiled
  for NVIDIA SM86 GPUs.
• SHA256SUMS: SHA-256 checksums for the four executables.

Downloaded executables may need chmod +x <filename> before use. Model weights
are not included.

The Darwin binaries target macOS 14.0 or newer and are ad-hoc signed. The Linux
binaries were built on Ubuntu 24.04 and dynamically use system libcurl. The
CUDA binary additionally requires CUDA 12 libraries and an NVIDIA driver; it
was built with CUDA 12.9 and validated on an RTX 3090.

Metal Build

make metal now compiles all .metal translation units into one metallib and
automatically embeds it in the final Mach-O executable. No colocated
embeddinggemma.metallib file is needed at runtime. EI_METALLIB_PATH remains
available as a diagnostic override.

Validation

• CPU, Metal, and CUDA pass all 10 llama.cpp golden embeddings.
• CPU, Metal, and CUDA pass packed-batch parity.
• HTTP Matryoshka dimensions 768, 512, 256, and 128 pass using every stripped
  release executable.
• Metal passes the full backend parity suite without a metallib sidecar.
• CUDA attention, QKV epilogue, Q4 MMA, and Q8 latency route parity pass.

embeddinggemma.c v0.2.0

Historical CUDA source release, superseded by
v0.2.1.

The old CUDA binary tarball asset was removed. Current CPU, Metal, and CUDA
binaries are published directly as raw executables in v0.2.1. GitHub's
automatic source .tar.gz and .zip archives remain available for this tag.

embeddinggemma.c v0.1.0

Historical source release, superseded by
v0.2.1.

The old binary tarball assets were removed. Current CPU, Metal, and CUDA
binaries are published directly as raw executables in v0.2.1. GitHub's
automatic source .tar.gz and .zip archives remain available for this tag.