v1.11.0 — the embedder-coverage release

aikit v1.11.0 — the embedder-coverage release

1.10.x was about hardening what already shipped. 1.11.0 is about reach. It takes aikit's set of embedding models certified against their HuggingFace references from two to eight — and, just as importantly, makes "which models are supported" a fact the code proves rather than a line in a README.

The unlock underneath the model count is a pure-Go SentencePiece/Unigram tokenizer (no cgo, no sentencepiece dependency), which opens the entire XLM-RoBERTa multilingual family, plus a mixture-of-experts FFN for the one MoE embedder. Everything is certified the same way: cosine 1.000000 against the reference, a hidden-state gate, and a break-it-first check that must go red when you perturb the pooling, the position offset, or the expert routing.

This is a minor release — no breaking changes. New API and new capability only; every behavior change widens what loads (things that used to fail now succeed).

---

Certified embedders (2 → 8)

Each row is gated by a parity test at cosine ≥ 0.9999 (all currently at 1.000000):

Model	Architecture	Pooling	Tokenizer	Dims	Truncatable
all-MiniLM-L6-v2	BERT	mean	WordPiece	384	—
CodeRankEmbed	nomic-bert (RoPE, SwiGLU)	cls	WordPiece	768	—
bge-small-en-v1.5	BERT	cls	WordPiece	384	—
nomic-embed-text-v1.5	nomic-bert (RoPE, SwiGLU)	mean	WordPiece	768	768→64
xlm-roberta-base	XLM-R	— (bare LM)	Unigram	768	—
multilingual-e5-base	XLM-R	mean	Unigram	768	—
bge-m3	XLM-R	cls	Unigram	1024	—
nomic-embed-text-v2-moe	nomic-bert + MoE (top-2/8)	mean	Unigram	768	768→256

Bold rows are new in this release. The generated docs/embedder-coverage.md is the source of truth.

---

Highlights

Pure-Go SentencePiece/Unigram tokenizer. A Viterbi-decoded Unigram model plus its normalizer and pre-tokenizer, all in Go — the piece that makes the multilingual family work end-to-end. No cgo boundary, no external tokenizer library.

Mixture-of-experts. Top-2-of-8 routing on alternating layers, certifying nomic-embed-text-v2-moe — the first MoE encoder in aikit.

encoder.MatryoshkaFloor(model) (min int, ok bool) — the one capability a serve layer must not guess. Truncating a Matryoshka-trained embedding to a shorter width is fine; truncating any other model returns a unit-length, entirely plausible vector that simply retrieves worse — a silent failure. This exports the per-model floor so a dimensions request can be refused instead of silently degraded. Only two of the eight certified models qualify, and the claim is measured, not asserted: at a quarter width, multilingual-e5-base drops paraphrase-pair recall 1.00 → 0.80 while genuine MRL models hold their floor.

Coverage you can't fake. docs/embedder-coverage.md is generated from a registry whose pooling/dimension claims are read back from the real checkpoints, behind a freshness gate — so the published table can't drift from the code, and a model can't be listed without a passing gate behind it.

Declared pooling, explicit loader variants. Pooling (CLS vs mean) is now read from 1_Pooling/config.json rather than assumed — the difference is silent and total when wrong. BERT loaders gained the XLM-R position-id offset (pad+1) and optional token_type embeddings.

---

Upgrading

go get github.com/townsendmerino/aikit@v1.11.0

Drop-in. The changed behaviors only accept more:

• LoadBERT no longer hard-fails on a tokenizer it can't parse — the model loads forward-only (best-effort) so you can still run it on pre-tokenized ids.
• embed.LoadTokenizer now accepts Unigram, not only WordPiece.
• (*encoder.Config).ValidateAssumptions accepts configs it used to reject (the new loader variants made them legitimate).
• (*embed.Tokenizer).EncodeWithSpecials reads the tokenizer's post-processor template instead of hardcoding [CLS]/[SEP].

If you serve an OpenAI-style dimensions parameter, this is the release to wire encoder.MatryoshkaFloor into your request validation.

---

Quality

Cut after a full qualification sweep: gofmt / vet / golangci-lint clean; the complete suite and go test -race ./... green; the chunk/treesitter cgo submodule green; a fuzz smoke pass over every untrusted-input parser; and apidiff confirming the Hard-tier 1.0 compatibility guarantee holds (every exported change is additive). CI covers Linux (amd64 + arm64, with the NEON asm and bit-identity gates) and Windows; the sweep additionally ran the full race suite natively on arm64/macOS.

Full changelog: CHANGELOG.md · Compare: v1.10.1...v1.11.0