eeANE 1.5.0

eeANE 1.5.0 adds a fourth architecture family, and it is the first one that is not an encoder. Qwen3 embedding models and Qwen3 generative rerankers now compile and serve through the ordinary eeane compile / eeane serve path, bringing the verified catalogue from 60 to 62 models.

A decoder as a text encoder

Every family eeANE supported until now reads a sentence in both directions and builds its vector by averaging over all positions or by taking a leading classification token. A decoder-style embedding model reads left to right, so only its last real token has seen the whole sentence — that is the position its vector comes from.

Two things about that architecture decide how it converts. Its attention mask has to combine causality with padding, and its grouped attention heads have to be expanded before the attention product. The conversion builds the mask itself, as a four-dimensional additive tensor handed straight to the backbone, which bypasses the framework's own mask construction while computing the same numbers — so no mask helper has to be rewritten, and the fp32 reference still goes through the framework's own path as an independent check. The head expansion is rewritten to stay four-dimensional, because the five-dimensional intermediate upstream produces makes the Neural Engine compiler reject the attention subgraph and fall the entire model back to the CPU.

On an M2 Mac mini, Qwen/Qwen3-Embedding-0.6B compiles to all three default buckets with 99.4% of its operations on the Neural Engine, at 18.8 ms for 128 tokens and 102.4 ms for 512. Its output agrees with sentence-transformers running the stock checkpoint at cosine 0.9998 or better, on queries, on documents, and across sixty corpus paragraphs.

Telling an embedding model from a reranker

eeane compile has always read the model kind from the architecture name in config.json: a name ending in ForSequenceClassification is a reranker, ...Model an embedding model. Decoder-style checkpoints break that rule, because an embedding model and a reranker of this family are both published as Qwen3ForCausalLM.

For a name ending in ForCausalLM, and only for those, the compiler now reads the model directory's sentence-transformers module declaration instead: a pooling module means an embedding model, a scoring module means a reranker. A directory declaring neither is refused with an error naming --kind as the fix, rather than compiled on a guess. Detection for every other architecture is untouched.

A reranker with no scoring head

A generative reranker does not classify a pair. It embeds the query and the document in a fixed chat-style instruction and reads its verdict from two vocabulary logits — "yes" and "no" — at the final position.

The compiled graph emits a single number, the difference between those two logits, because the relevance of a two-way choice satisfies

softmax([no, yes])[yes] = sigmoid(yes - no)

That difference is therefore exactly what /rerank's existing sigmoid scoring already turns into a probability, so the API does not change in any way: a caller cannot tell a generative reranker from a cross-encoder one, and raw_scores=true returns the raw difference just as it returns a raw logit elsewhere. Keeping only two rows of the output projection also leaves a matrix of six figures' worth of rows out of the compiled model entirely.

The instruction template a model wants its pairs spelled out with is resolved once at compile time from the model's own declaration, recorded in the artifact, and applied by the server on every request. When a pair does not fit its bucket, only the body is truncated — the preamble and the closing suffix are always kept, because the suffix is where the template asks the model to answer.

Qwen/Qwen3-Reranker-0.6B compiles to both default buckets with 99.3% Neural Engine placement. Against CrossEncoder on the stock checkpoint it identifies the most relevant document for all nine test queries with no ranking disagreement, and its served token sequence is byte-identical to what the reference implementation feeds the model.

Judging a reranker on the right scale

The compile-time self-check compares a reranker's compiled scores against an fp32 reference. It used to do so after the sigmoid, which suits a cross-encoder: that architecture emits one calibrated value per pair, and the probability is the number the server hands out.

A generative reranker's score is a difference of two raw logits, spread over a much wider range that the sigmoid squashes unevenly — its slope peaks in the middle and falls away to nothing at either end. Measured after the sigmoid, the same conversion error would count for a great deal on a pair that happens to land near the middle and almost nothing on one at the edge, so a model would pass or fail on where its fixtures happen to sit rather than on how faithfully it was converted. A backend now declares which space its scores are faithful in, and the self-check measures accordingly. The cross-encoder path keeps the space it always had, down to the keys it records.

Notes

Compiled weights are memory-mapped rather than copied per bucket, so serving several buckets of one model does not multiply its resident memory: all three buckets of a 0.6B-class model add about 150 MB to the server process, and an eight-model configuration holds workers at 592 MB.

The larger models of this family are not supported. Past roughly 2 GiB of compiled model, the Neural Engine stops accepting any operation and everything runs on the CPU instead — conversion, loading and inference all still succeed, so the only symptom is that it is slow. In parameter count that ceiling sits near 1.07B.

Some embedding models, including this one, expect an instruction string in front of the search query. eeANE does not alter request text and still does not: prepending it is the caller's job. The README's API section gives the format this model publishes and how to configure a common client to send it.

Previously verified models are unaffected: re-compiling them reproduces their recorded accuracy numbers and operation counts exactly, and artifacts written by earlier releases still load.