eeANE 1.6.0

eeANE 1.6.0 adds a fifth architecture family: the bidirectional embedding models of Gemma 3. google/embeddinggemma-300m now compiles and serves through the ordinary eeane compile / eeane serve path, bringing the verified catalogue from 62 to 63 models. The server is unchanged.

Two kinds of layer, two masks

The layers of this architecture come in two types. Some let a position see every real token; the others only the keys within a window of it, on either side. The conversion builds one additive mask for each type inside the graph and hands both to the backbone keyed by layer type, so each layer takes its own, while the fp32 reference still goes through the framework's own mask construction as an independent check.

Handing every layer one shared mask — the way a causal decoder is masked — would silently drop the window and compute a different model for any sequence longer than it. Nothing shows up on a short sentence, which is why agreement with the original model is measured on long texts too.

Only the bidirectional embedding model is compiled. The same architecture configured causally is a language model, so a directory that does not declare bidirectional attention is refused before any weight is read, as are the multimodal model and Gemma 3n.

A model that does not fit float16

The upstream model card states that this model's activations do not fit float16, and a compiled model runs in float16. Converted as it stands, it agrees with its fp32 self only to a cosine of 0.98. Three things are behind that, and each is removed by a rewrite that leaves the fp32 function unchanged:

• The residual stream grows past the largest float16 number in the later layers. It is divided by a power of two; the normalizations that read the stream are scale-invariant, so nothing else changes.
• The output of the later MLPs becomes so small that its square falls below the smallest normal float16 number, which the Neural Engine does not resolve. Each layer's output projection is multiplied by a power of two of its own, and the normalization that follows undoes it.
• From a sequence length of 1024 on, the Neural Engine returns wrong numbers for an attention whose per-head score matrix reaches 2^20 elements, although the CPU computes the same program correctly: 1023 is right, 1024 is not. Attention over such a sequence is computed in ranges of query rows.

The factors are measured on the loaded model at conversion time and recorded in the artifact metadata. Only the compiled copy is rewritten, always and without a flag.

Measured

On an M2 Mac mini (16 GB, macOS 27.0.1), all three default buckets pass the self-check with 99.2–99.3% of their operations on the Neural Engine, at 5.1 ms for 128 tokens, 25.0 ms for 512 and 68.9 ms for 1024.

Against sentence-transformers running the stock checkpoint on the CPU in fp32:

Condition	Cosine min
Short texts (en / ja / zh)	0.99914
Query prefix, concatenated by the client vs. prompt_name="query"	0.99930
Document prefix, concatenated by the client vs. prompt_name="document"	0.99941
60 paragraphs, bucket 512	0.99841
Long texts, bucket 512 / 1024	0.99845 / 0.99821

The same conversion run in fp32 agrees with the reference at 0.9999998 or better, so what remains is the arithmetic precision of float16 and the Neural Engine, not the conversion.

End to end at batch 1, eeANE takes 7.6 / 27.3 / 70.4 ms for 128 / 512 / 1024 tokens — 7.0–8.9 times faster than PyTorch on the CPU and 3.5–10.9 times faster than PyTorch on MPS. The README has the full tables and conditions.

Prefixes stay with the client

EmbeddingGemma expects task: search result | query: in front of a search query and title: none | text: in front of a document. eeANE does not alter the text it is given, so the client adds them, as with Qwen3. The measurements above show that doing so gives the same result as the model's own prompts. The README lists the settings for Open WebUI.

Models that need their terms accepted

google/embeddinggemma-300m can only be downloaded after its terms have been accepted on Hugging Face. When a download is refused for that reason, or because a login is missing, the error now adds a line naming the remedy. eeANE reads no credential; that stays with the Hugging Face libraries. eeANE converts models on your machine and distributes no weights, so each model's license and terms remain yours to follow.

Notes

• Qwen3 on macOS 27.0.1. On macOS 27.0.1 the two Qwen3 models are no longer placed on the Neural Engine and run on the CPU; on macOS 26.6.2, 99.3–99.4% of their operations were placed. The same holds for the 1.5.0 code and for untouched 1.5.0 artifacts, so it is not caused by this release. Results stay correct; inference is slower. The other four families are placed as before. The cause is under investigation.
• The 2048 bucket of EmbeddingGemma compiles but runs on the CPU. Stay with the default 128 / 512 / 1024.
• Long texts agree with the original model at 0.998 rather than the 0.999 reached on short ones.
• No change to the runtime, the configuration format or the artifact format; existing artifacts load as before.