v0.3.0 — Faster than llama.cpp on every backend

embeddinggemma.c v0.3.0

Faster than llama.cpp across the board, on every backend.

This release lands a batch-scheduler overhaul that lifts throughput on CPU,
Metal, CUDA, ROCm, and XPU, plus new Metal 4 kernels and a serving-path
rewrite. Measured over HTTP against llama.cpp b8981 on an Apple M5 Max —
identical GGUF, exact token counts, caches disabled, fresh servers per cell —
embeddinggemma is faster in all 54 shapes, geometric mean 1.25x,
range 1.01x–2.01x.

Highlights

• Demand-aware wave batch collection. The dynamic batcher now tracks peak
  in-flight demand and dispatches on wave completion instead of a fixed timer,
  fixing an alternating-singleton pathology that suppressed batching at
  concurrency 2 on every backend. Measured lift at c2: CPU +18%, CUDA +78%,
  ROCm +64%, XPU +55%. Wins of +15–35% across higher concurrency.
• Metal 4 tensor projections and flash attention (Apple GPUs reporting
  MTLGPUFamilyMetal4), embedded alongside the Metal 3.1 library with runtime
  family selection so older Apple Silicon keeps working.
• Serving-path rewrite: Ryu shortest-round-trip float formatting (~3x
  faster responses), single-writev responses with TCP_NODELAY,
  SO_RCVTIMEO keep-alive, and split Metal command encoding for CPU/GPU
  overlap.
• Correctness: a single-flight fix for concurrent inline/backend execution;
  validated across all backends (min cross-backend cosine ≥ 0.999, 1,000,012
  exact float round-trips).

Fixes

• Linux build: include <sys/time.h> for struct timeval in the keep-alive
  path (every Linux target failed to build under -Werror otherwise).
• README examples now use the correct EmbeddingGemma prompt prefixes
  (task: search result | query: / title: none | text:).

Assets

Raw executables, one per platform/backend. Verify with checksums.txt.

platform	backend	asset
macOS ARM64	CPU	embeddinggemma-darwin-arm64-cpu
macOS ARM64	Metal	embeddinggemma-darwin-arm64-metal
Linux x86_64	CPU	embeddinggemma-linux-x86_64-cpu
Linux x86_64	CUDA	embeddinggemma-linux-x86_64-cuda
Linux x86_64	ROCm	embeddinggemma-linux-x86_64-rocm
Linux x86_64	XPU SYCL	embeddinggemma-linux-x86_64-xpu