Skip to content

aikit v1.6.0 — the GEMM release: public matmul 7% → 69% of peak

Choose a tag to compare

@github-actions github-actions released this 12 Jun 18:18

A measurement-driven performance release. An external review suggested the
f32 matmul formulation had register-level headroom; the house rule is
measure first, so this cycle started with a peak-fraction gate — and the
gate found more than the review predicted. Three changes later, the public
linalg.MatmulBT went from ~7% to 46–69% of the machine's measured f32
peak
(~6.3× at prefill shapes), with every consumer inheriting it. Plus
one small Hard-tier addition from the goinfer cross-repo review.

The GEMM campaign (arm64; AVX2 ports are hardware-gated follow-ups)

  • Measured the ceiling before touching anything. A register-saturating
    FMA micro-benchmark clocked the M1 Pro P-core at 95.4 GFLOPS f32
    (≈15 FMA/cycle — correcting the 8/cycle back-of-envelope). Against that
    measured ceiling, the encoder's blocked GEMM read 40–49% — below the
    ≤50% gate, so the work was justified by numbers, not vibes.
    (BenchmarkGEMMPeakFraction is now in the tree.)
  • Dot2x8 — a 2×8 register micro-kernel replaces the 1×8 Dot8x4 as
    the blocked GEMM's inner loop: 2 a-rows × 8 b-rows, 16 accumulators held
    across the K loop, every b-load feeding two FMLAs. Same accumulation
    order per dot, so it's bit-identical — golden parity unchanged, no
    tolerance changes. Encoder FC matmuls 1.52–1.58×; end-to-end
    EncodeBatch 1.36×, single-doc encode 1.27×. M=1 decode/gemv
    paths untouched (they're bandwidth-bound; nothing to win).
  • The blocked GEMM is now linalg.MatmulBT itself, not an encoder
    private.
    A cross-repo gate (goinfer prefill) caught the public
    MatmulBT running a naive re-streaming span at ~7% of peak — every
    consumer outside the encoder was paying for the missing cache blocking.
    The encoder's blocked+register-tiled implementation is hoisted into
    linalg as the single shared home. Prefill shapes: 7% → 46%
    (~6.3×); transformer tiles 68–75%. Small matmuls keep the naive span
    via a threshold, so nothing regresses.
  • B-panel packing at large K fixes L1 set-associativity conflicts
    (8 b-rows K·4 bytes apart collide; packing makes them contiguous).
    Bit-identical — same values, same order. Prefill M=512×4096×4096:
    46% → 69% of peak; the encoder's own K=3072 fc2 +15%. Gated to
    K≥2048; the K=768 tiles were already conflict-free.

Also added

  • linalg.MatmulBTInto — the serial blocked GEMM into a caller-owned
    dst, for consumers that manage their own parallelism (Experimental).
  • embed.SafetensorsFile.TensorF32 / TensorI32 — shape-checked typed
    tensor reads with BF16/F16 widening (Hard-tier, additive). Lifts the
    helper that aikit's own loaders and goinfer's decoder/vision loaders had
    hand-written ≥6 times between them; goinfer drops its copies at its next
    pin bump. Direct outcome of the 2026-06-12 cross-repo review.

Numerics & compatibility

  • MatmulBT's results differ from the old naive order by ~1e-5 — f32
    reassociation from the blocked accumulation order, the same accepted
    class as the v1.2.0 ann change. Its documented width-invariance is
    preserved
    (column shards align to the 8-column kernel group;
    TestParallelWidth_bitIdentical). Callers needing exact determinism
    have MatmulBTAcc64, unchanged.
  • Dot2x8 and the B-panel packing are bit-identical to their
    predecessors; encoder golden parity is unchanged across the entire
    campaign.
  • Hard tier: additive only (TensorF32/TensorI32). All new linalg
    surface is Experimental. -race clean; cgo-free; Windows cross-build
    verified.
  • The 2×8 kernel and packing are arm64 NEON; amd64 keeps its AVX2 path —
    the ports are tracked, gated on x86 test hardware (roadmap §2.4).

Full detail: CHANGELOG.md