eeANE 1.6.1

eeANE 1.6.1 brings the Qwen3 models back onto the Neural Engine on macOS 27, where the size of model the Neural Engine accepts was found to have dropped from 2 GiB to 1 GiB. If you serve Qwen3 models on macOS 27 or later, recompile them with this version.

What changed in macOS 27

Up to macOS 26, the Neural Engine accepted a compiled model of up to about 2 GiB. From macOS 27 on, the limit is 1 GiB (1,073,741,824 bytes). A model above it is not rejected: conversion, loading and inference all succeed and the system logs nothing. Every operation of the model simply runs on the CPU.

The limit was located with synthetic models of one shape that differ only in vocabulary size: 1,072,681,458 bytes is placed on the Neural Engine (99.6% of its operations) and 1,074,770,418 bytes is not placed at all. It does not depend on the architecture. This was measured on an M2 Mac mini with macOS 27.0.1; Apple has not documented the change, so it may differ on other hardware or change again.

The two Qwen3 models compile to 1.11 GiB. On macOS 27 they kept returning correct results, about 3.5 times slower.

Recompile, and the model fits again

eeane compile Qwen/Qwen3-Embedding-0.6B
eeane compile Qwen/Qwen3-Reranker-0.6B

No option is needed: artifacts made by an earlier eeANE are rebuilt automatically. When a Qwen3 model compiled on macOS 27 or later exceeds the limit, eeane compile stores its vocabulary embedding table as 8-bit integers — every other weight stays FP16 — and compiles again. That table is read one row per token, so little accuracy is at stake, and it is about a quarter of the model.

Measured on an M2 Mac mini (16 GB, macOS 27.0.1):

Model	Bucket	Neural Engine placement	Warm latency	vs CPU-only
Qwen/Qwen3-Embedding-0.6B	128	99.4%	19.3 ms	3.5x
Qwen/Qwen3-Embedding-0.6B	512	99.4%	104.2 ms	2.5x
Qwen/Qwen3-Embedding-0.6B	1024	99.4%	287.6 ms	2.3x
Qwen/Qwen3-Reranker-0.6B	512	99.3%	105.5 ms	2.4x
Qwen/Qwen3-Reranker-0.6B	1024	99.3%	288.0 ms	2.3x

Placement and speed match what these models showed on macOS 26. The built-in self-check is unchanged (cosine 0.99994 for the embedding model). Against the reference implementations, the embedding model's row-wise cosine stays at 0.9996 or higher and the reranker's score differs by at most 0.23 in logit space, with the top result identical for every query tested; among 567 document pairs whose reference scores differ by more than 0.2, one pair of clearly irrelevant documents changed order.

Nothing else is touched. A model under the limit, a model of any other architecture, and any model compiled on macOS 26 or earlier come out exactly as before.

The operating system you compile on decides

The reduction is applied only when eeane compile runs on macOS 27 or later. So after upgrading a Mac to macOS 27, recompile the affected models: artifacts made on the older system are above the new limit. Behaviour on macOS 26 or earlier is covered by tests, not by a run on such a system.

The server now says so

eeane serve checks every configured model at start-up and logs a warning for one that will not run on the Neural Engine — because that was already recorded when it was compiled, or because it is above 1 GiB on macOS 27 or later. The server still starts; the warning names the model, the buckets, the reason and what to run.

Not fixed yet: XLM-RoBERTa large-class models

The size reduction covers the Qwen3 family only. The XLM-RoBERTa large-class models in the verified list are above 1 GiB as well and cannot currently be used on macOS 27 or later: intfloat/multilingual-e5-large was measured (0% placement, and an output that is not a number), and BAAI/bge-m3, Snowflake/snowflake-arctic-embed-l-v2.0, intfloat/multilingual-e5-large-instruct, BAAI/bge-reranker-v2-m3, BAAI/bge-reranker-large and hotchpotch/japanese-bge-reranker-v2-m3-v1 share its size and are expected to behave the same. They keep working on macOS 26 and earlier. Support is planned for a later release.

The README has a section on all of this, linked from its top: "macOS 27 and the 1 GiB Neural Engine limit".

eeANE 1.6.0

eeANE 1.6.0 adds a fifth architecture family: the bidirectional embedding models of Gemma 3. google/embeddinggemma-300m now compiles and serves through the ordinary eeane compile / eeane serve path, bringing the verified catalogue from 62 to 63 models. The server is unchanged.

Two kinds of layer, two masks

The layers of this architecture come in two types. Some let a position see every real token; the others only the keys within a window of it, on either side. The conversion builds one additive mask for each type inside the graph and hands both to the backbone keyed by layer type, so each layer takes its own, while the fp32 reference still goes through the framework's own mask construction as an independent check.

Handing every layer one shared mask — the way a causal decoder is masked — would silently drop the window and compute a different model for any sequence longer than it. Nothing shows up on a short sentence, which is why agreement with the original model is measured on long texts too.

Only the bidirectional embedding model is compiled. The same architecture configured causally is a language model, so a directory that does not declare bidirectional attention is refused before any weight is read, as are the multimodal model and Gemma 3n.

A model that does not fit float16

The upstream model card states that this model's activations do not fit float16, and a compiled model runs in float16. Converted as it stands, it agrees with its fp32 self only to a cosine of 0.98. Three things are behind that, and each is removed by a rewrite that leaves the fp32 function unchanged:

• The residual stream grows past the largest float16 number in the later layers. It is divided by a power of two; the normalizations that read the stream are scale-invariant, so nothing else changes.
• The output of the later MLPs becomes so small that its square falls below the smallest normal float16 number, which the Neural Engine does not resolve. Each layer's output projection is multiplied by a power of two of its own, and the normalization that follows undoes it.
• From a sequence length of 1024 on, the Neural Engine returns wrong numbers for an attention whose per-head score matrix reaches 2^20 elements, although the CPU computes the same program correctly: 1023 is right, 1024 is not. Attention over such a sequence is computed in ranges of query rows.

The factors are measured on the loaded model at conversion time and recorded in the artifact metadata. Only the compiled copy is rewritten, always and without a flag.

Measured

On an M2 Mac mini (16 GB, macOS 27.0.1), all three default buckets pass the self-check with 99.2–99.3% of their operations on the Neural Engine, at 5.1 ms for 128 tokens, 25.0 ms for 512 and 68.9 ms for 1024.

Against sentence-transformers running the stock checkpoint on the CPU in fp32:

Condition	Cosine min
Short texts (en / ja / zh)	0.99914
Query prefix, concatenated by the client vs. prompt_name="query"	0.99930
Document prefix, concatenated by the client vs. prompt_name="document"	0.99941
60 paragraphs, bucket 512	0.99841
Long texts, bucket 512 / 1024	0.99845 / 0.99821

The same conversion run in fp32 agrees with the reference at 0.9999998 or better, so what remains is the arithmetic precision of float16 and the Neural Engine, not the conversion.

End to end at batch 1, eeANE takes 7.6 / 27.3 / 70.4 ms for 128 / 512 / 1024 tokens — 7.0–8.9 times faster than PyTorch on the CPU and 3.5–10.9 times faster than PyTorch on MPS. The README has the full tables and conditions.

Prefixes stay with the client

EmbeddingGemma expects task: search result | query: in front of a search query and title: none | text: in front of a document. eeANE does not alter the text it is given, so the client adds them, as with Qwen3. The measurements above show that doing so gives the same result as the model's own prompts. The README lists the settings for Open WebUI.

Models that need their terms accepted

google/embeddinggemma-300m can only be downloaded after its terms have been accepted on Hugging Face. When a download is refused for that reason, or because a login is missing, the error now adds a line naming the remedy. eeANE reads no credential; that stays with the Hugging Face libraries. eeANE converts models on your machine and distributes no weights, so each model's license and terms remain yours to follow.

Notes

• Qwen3 on macOS 27.0.1. On macOS 27.0.1 the two Qwen3 models are no longer placed on the Neural Engine and run on the CPU; on macOS 26.6.2, 99.3–99.4% of their operations were placed. The same holds for the 1.5.0 code and for untouched 1.5.0 artifacts, so it is not caused by this release. Results stay correct; inference is slower. The other four families are placed as before. The cause is under investigation.
• The 2048 bucket of EmbeddingGemma compiles but runs on the CPU. Stay with the default 128 / 512 / 1024.
• Long texts agree with the original model at 0.998 rather than the 0.999 reached on short ones.
• No change to the runtime, the configuration format or the artifact format; existing artifacts load as before.

eeANE 1.5.0

eeANE 1.5.0 adds a fourth architecture family, and it is the first one that is not an encoder. Qwen3 embedding models and Qwen3 generative rerankers now compile and serve through the ordinary eeane compile / eeane serve path, bringing the verified catalogue from 60 to 62 models.

A decoder as a text encoder

Every family eeANE supported until now reads a sentence in both directions and builds its vector by averaging over all positions or by taking a leading classification token. A decoder-style embedding model reads left to right, so only its last real token has seen the whole sentence — that is the position its vector comes from.

Two things about that architecture decide how it converts. Its attention mask has to combine causality with padding, and its grouped attention heads have to be expanded before the attention product. The conversion builds the mask itself, as a four-dimensional additive tensor handed straight to the backbone, which bypasses the framework's own mask construction while computing the same numbers — so no mask helper has to be rewritten, and the fp32 reference still goes through the framework's own path as an independent check. The head expansion is rewritten to stay four-dimensional, because the five-dimensional intermediate upstream produces makes the Neural Engine compiler reject the attention subgraph and fall the entire model back to the CPU.

On an M2 Mac mini, Qwen/Qwen3-Embedding-0.6B compiles to all three default buckets with 99.4% of its operations on the Neural Engine, at 18.8 ms for 128 tokens and 102.4 ms for 512. Its output agrees with sentence-transformers running the stock checkpoint at cosine 0.9998 or better, on queries, on documents, and across sixty corpus paragraphs.

Telling an embedding model from a reranker

eeane compile has always read the model kind from the architecture name in config.json: a name ending in ForSequenceClassification is a reranker, ...Model an embedding model. Decoder-style checkpoints break that rule, because an embedding model and a reranker of this family are both published as Qwen3ForCausalLM.

For a name ending in ForCausalLM, and only for those, the compiler now reads the model directory's sentence-transformers module declaration instead: a pooling module means an embedding model, a scoring module means a reranker. A directory declaring neither is refused with an error naming --kind as the fix, rather than compiled on a guess. Detection for every other architecture is untouched.

A reranker with no scoring head

A generative reranker does not classify a pair. It embeds the query and the document in a fixed chat-style instruction and reads its verdict from two vocabulary logits — "yes" and "no" — at the final position.

The compiled graph emits a single number, the difference between those two logits, because the relevance of a two-way choice satisfies

softmax([no, yes])[yes] = sigmoid(yes - no)

That difference is therefore exactly what /rerank's existing sigmoid scoring already turns into a probability, so the API does not change in any way: a caller cannot tell a generative reranker from a cross-encoder one, and raw_scores=true returns the raw difference just as it returns a raw logit elsewhere. Keeping only two rows of the output projection also leaves a matrix of six figures' worth of rows out of the compiled model entirely.

The instruction template a model wants its pairs spelled out with is resolved once at compile time from the model's own declaration, recorded in the artifact, and applied by the server on every request. When a pair does not fit its bucket, only the body is truncated — the preamble and the closing suffix are always kept, because the suffix is where the template asks the model to answer.

Qwen/Qwen3-Reranker-0.6B compiles to both default buckets with 99.3% Neural Engine placement. Against CrossEncoder on the stock checkpoint it identifies the most relevant document for all nine test queries with no ranking disagreement, and its served token sequence is byte-identical to what the reference implementation feeds the model.

Judging a reranker on the right scale

The compile-time self-check compares a reranker's compiled scores against an fp32 reference. It used to do so after the sigmoid, which suits a cross-encoder: that architecture emits one calibrated value per pair, and the probability is the number the server hands out.

A generative reranker's score is a difference of two raw logits, spread over a much wider range that the sigmoid squashes unevenly — its slope peaks in the middle and falls away to nothing at either end. Measured after the sigmoid, the same conversion error would count for a great deal on a pair that happens to land near the middle and almost nothing on one at the edge, so a model would pass or fail on where its fixtures happen to sit rather than on how faithfully it was converted. A backend now declares which space its scores are faithful in, and the self-check measures accordingly. The cross-encoder path keeps the space it always had, down to the keys it records.

Notes

Compiled weights are memory-mapped rather than copied per bucket, so serving several buckets of one model does not multiply its resident memory: all three buckets of a 0.6B-class model add about 150 MB to the server process, and an eight-model configuration holds workers at 592 MB.

The larger models of this family are not supported. Past roughly 2 GiB of compiled model, the Neural Engine stops accepting any operation and everything runs on the CPU instead — conversion, loading and inference all still succeed, so the only symptom is that it is slow. In parameter count that ceiling sits near 1.07B.

Some embedding models, including this one, expect an instruction string in front of the search query. eeANE does not alter request text and still does not: prepending it is the caller's job. The README's API section gives the format this model publishes and how to configure a common client to send it.

Previously verified models are unaffected: re-compiling them reproduces their recorded accuracy numbers and operation counts exactly, and artifacts written by earlier releases still load.

eeANE 1.4.0

eeANE 1.4.0 closes two gaps in what eeane compile can faithfully reproduce — sentence-transformers Dense projection modules and plain-RoBERTa architectures — and one gap in how it judges its own work: the accuracy self-check no longer measures every model against fixtures in a single fixed language. It also adds the OpenAI-compatible dimensions parameter to /v1/embeddings, and grows the verified catalogue from 51 to 60 models.

The self-check now speaks three languages

The compile-time accuracy check compares the compiled fp16 artifact against an fp32 reference on a small fixed set of sentences. Until now that set came in one language per backend — English for BERT, Japanese for the others, an accident of which model family each backend was born for. For a model whose vocabulary barely covers that language, the fixtures encode to little more than unknown-token fragments, and the amplified fp16-vs-fp32 difference reflects the input rather than the model: an English-only ModernBERT model could fail the check without anything being wrong with it.

eeane compile now scores three fixed sets — English, Japanese and Chinese — and accepts a variant as soon as any one of them clears the unchanged threshold. The log names every set's result and the one the variant was accepted on, and the same numbers are recorded in the artifact metadata:

      s512: sanity : pass (best=en 0.99883; ja 0.98737, zh 0.99236)

The existing fixtures were inherited word for word, so accuracy numbers recorded for previously verified models remain directly comparable.

Dense projection modules are compiled in

Some sentence-transformers models apply a linear projection after pooling — declared as a Dense module in the model directory's modules.json — and their published embeddings only exist on the far side of it. Earlier releases did not read that file, so such a model compiled without its projection: the self-check still passed, because both sides of it made the same omission, while the served vectors disagreed with the published model in both meaning and width.

eeane compile now reads the declared module chain, supports Transformer -> Pooling -> Dense (any number) -> Normalize (optional, trailing), and bakes each Dense stage (Identity or Tanh activation) into both the compiled graph and the fp32 reference — the embedding width follows the projection, so mdbr-leaf-mt correctly serves 1024-dimensional vectors from a 384-wide backbone. A chain the conversion cannot reproduce is refused before any weight is read, instead of silently compiling something with a different meaning. Dense checkpoints go through the same safetensors-first policy as the main weights, the declaration is recorded in the artifact metadata, and a changed declaration triggers reconversion. The trailing Normalize module is not baked in: the server applies L2 normalization according to each model's normalize setting, as before.

RoBERTa architectures

Models whose config.json declares a RobertaModel architecture are now routed to the XLM-RoBERTa backend. transformers implements RoBERTa and XLM-RoBERTa as the same encoder, differing only in vocabulary, so the existing backend — position offsets included — applies unchanged.

Truncatable embeddings

/v1/embeddings accepts the OpenAI-compatible dimensions parameter: the embedding is truncated to its first dimensions components and then re-normalized (when normalization is enabled), returning a smaller vector without re-running inference. This is meaningful for models trained with a Matryoshka representation learning objective, whose embeddings stay useful at any prefix length. Requesting more dimensions than the model's embedding width returns a 400.

Nine newly verified models

Each compiles with a single eeane compile <HF ID>, passed its self-check with unchanged thresholds, and had its output compared against the reference sentence-transformers / CrossEncoder implementation on an M2 Mac (embedding per-text cosine 0.99994–0.99999, reranker sigmoid difference at most 0.0013, Neural Engine placement 90.7–98.1%):

• MongoDB/mdbr-leaf-mt — 23M, BERT with a Dense projection to 1024 dimensions
• MongoDB/mdbr-leaf-ir — 23M, Dense projection to 768 dimensions
• sentence-transformers/LaBSE — 471M, 109-language, CLS pooling with a Tanh Dense head
• ibm-granite/granite-embedding-30m-english — 30M, RobertaModel
• ibm-granite/granite-embedding-125m-english — 125M, RobertaModel
• nomic-ai/modernbert-embed-base — 149M; failed 1.3.0's Japanese-only self-check (that measurement reproduces bit-for-bit under the new per-set report) and passes on the English set, where this English model belongs
• hotchpotch/japanese-reranker-cross-encoder-xsmall-v1 — 107M Japanese cross-encoder reranker
• hotchpotch/japanese-reranker-cross-encoder-small-v1 — 118M Japanese cross-encoder reranker
• hotchpotch/japanese-bge-reranker-v2-m3-v1 — 568M Japanese reranker built on bge-reranker-v2-m3

The Verified models table in the README lists all 60.

Verification

• 1136 tests pass, up from 961. The new ones cover per-set sanity evaluation and every way it can go wrong, the Dense declaration reader and its refusal paths, a full conversion of a synthetic Dense model, RoBERTa dispatch, and the dimensions parameter end to end.
• No regression: a recompiled cl-nagoya/ruri-v3-310m matches its pre-change comparison against sentence-transformers to the exact same values, and the default configuration passes its end-to-end server verification with a rerank median of 5.602 s, in line with previous releases.
• Eleven models were served at once, with each one's HTTP responses matched against direct Core ML inference (embedding cosine 1.00000000, reranker score difference 0.0), and dimensions was exercised over HTTP, base64 encoding included.

Install

uv tool install eeane

Full changelog: v1.3.0...v1.4.0

eeANE 1.3.0

eeANE 1.3.0 teaches the ModernBERT compile backend to read the pooling a model actually declares. Until now it baked mean pooling into every embedding graph, so a ModernBERT embedding model that uses CLS pooling was compiled with the wrong pooling — and, worse, it looked fine: the accuracy self-check compares the compiled artifact against eeANE's own reading of the same model, so both sides made the same mistake and agreed with each other. The BERT and XLM-RoBERTa backends had read the declaration since they were added; ModernBERT now does too.

If you serve ModernBERT models that use mean pooling (ruri-v3 and friends), nothing about their output changes — this was verified by reproducing the pre-change comparison against sentence-transformers to the eighth decimal place. What changes is that a family of models that could not be compiled correctly now can be.

Pooling is detected, not assumed

eeane compile reads 1_Pooling/config.json from the model directory before it loads any weights, compiles the matching graph, and prints which pooling it picked:

      kind / buckets  : embedding / 128, 512, 1024
      pooling         : cls

An embedding model that does not declare a supported pooling mode is now rejected with an error rather than compiled on a guess. A wrongly-pooled artifact returns plausible-looking vectors that mean something else, which is a worse outcome than a failed compile. If you hit this on a model you expect to work, check that it ships the sentence-transformers pooling module; Troubleshooting in the README covers it.

The resolved pooling is also recorded in the artifact metadata and in model_info.json, and a cached variant whose recorded pooling no longer matches what the model declares is reconverted instead of silently reused.

Rerankers are unaffected: their pooling belongs to the model's own classification head, not to a sentence-transformers module.

Five newly verified models

All five declare CLS pooling and could not be compiled correctly before this release. Each compiles with a single eeane compile <HF ID>, passed its self-check with unchanged thresholds, and had its output compared against the reference sentence-transformers implementation on an M2 Mac (per-text cosine 0.99976–0.99999869, Neural Engine placement 98.0–98.9%):

• ibm-granite/granite-embedding-small-english-r2 — 47M, 384 dimensions, about 2 ms warm for a 128-token request
• ibm-granite/granite-embedding-english-r2 — 149M, 768 dimensions
• ibm-granite/granite-embedding-97m-multilingual-r2 — 97M, 384 dimensions
• ibm-granite/granite-embedding-311m-multilingual-r2 — 311M, 768 dimensions
• Alibaba-NLP/gte-modernbert-base — 149M, 768 dimensions

The verified catalogue grows from 46 to 51 models. The Verified models table in the README lists them all.

Verification

• 961 tests pass, up from 928. The new ones cover pooling detection, every way the declaration can be unreadable, wrapper selection, the FP32 baseline following the detected pooling, and a full conversion of a synthetic CLS-pooling model.
• Seven previously verified ModernBERT models were recompiled: every embedding model resolved to mean pooling as declared, and the default configuration passes its end-to-end server verification with a rerank median of 5.629 s, in line with previous releases.
• Eight models were served at once, with each one's HTTP responses matched against direct Core ML inference (cosine 1.00000000).

Install

uv tool install eeane

Full changelog: v1.2.0...v1.3.0

eeANE 1.2.0

eeANE 1.2.0 is a verification release: 35 more models were run end to end and are now documented, bringing the verified total to 46. There are no engine changes — if you are already running eeANE, nothing about your setup changes. What is new is that many more models are now known to work, and there is a table telling you which.

35 newly verified models

Each of these compiles with a single eeane compile <HF ID> and serves through the existing multi-model server. Every one passed its accuracy self-check against the FP32 original with unchanged thresholds, and — new in this release — had its output compared against the reference sentence-transformers / CrossEncoder implementation on an M2 Mac: embedding cosine ≥ 0.9999, reranker scores within 0.0015 with matching top-1 and rank order. Neural Engine placement ranged from 91.2% to 98.9%.

Embedding models

• IBM granite — granite-embedding-107m-multilingual, granite-embedding-278m-multilingual
• Snowflake Arctic Embed — xs, s, m, m-v1.5, l, and l-v2.0 (the last one an 8192-token model that keeps the full 128/512/1024 bucket set)
• thenlper GTE — gte-base, gte-large
• mixedbread-ai/mxbai-embed-large-v1
• sentence-transformers — all-MiniLM-L6-v2, all-MiniLM-L12-v2, multi-qa-MiniLM-L6-cos-v1, paraphrase-MiniLM-L6-v2, paraphrase-multilingual-MiniLM-L12-v2, paraphrase-multilingual-mpnet-base-v2
• intfloat e5 — e5-small-v2, e5-base-v2, e5-large-v2, multilingual-e5-small, multilingual-e5-large-instruct
• cl-nagoya ruri-v3 — the small variants 30m, 70m, 130m (Japanese; a 128-token request takes about 1 ms warm on the 30m)
• hotchpotch/bekko-embedding-v1-a25m
• BAAI bge v1.5 Chinese — bge-small-zh-v1.5, bge-base-zh-v1.5, bge-large-zh-v1.5, plus the original bge-small-en

Rerankers

• ibm-granite/granite-embedding-reranker-english-r2
• hotchpotch japanese-reranker-v2 — tiny, xsmall, small, base (the tiny one scores a 512-token pair in about 2.3 ms warm)

A Verified models table

The README now has a Verified models section: one table per architecture family, giving each model's type and the buckets eeane compile produces for it. Worth knowing when you read it — a model's name does not always tell you which backend it uses. paraphrase-multilingual-mpnet-base-v2 is an XLM-RoBERTa model despite the name, and multilingual-e5-small is a BERT one while its larger siblings are not. The table groups by what config.json actually declares.

Models built on these three architectures that are not listed are likely to work as well; the table records what has been run end to end, not a limit.

Two limitations now documented

• CLS-pooling ModernBERT embedding models are not supported yet. The ModernBERT backend compiles mean pooling only, so an embedding model of that family declaring CLS pooling would be compiled with the wrong pooling. The BERT and XLM-RoBERTa backends read the declared pooling and handle both, and ModernBERT rerankers are unaffected. Support is planned.
• Accuracy on out-of-vocabulary input. Compiled models run in fp16. For input a model's tokenizer cannot represent well — feeding English text to a Chinese-only model, say — the rounding difference against an fp32 reference grows noticeably. Within a model's intended languages the agreement stays tight.

The Chinese bge v1.5 models, previously listed as unverified, are now verified. Two of them ship pytorch_model.bin only, so they need --allow-pickle.

Requires Python 3.11 or 3.12, an Apple Silicon Mac on macOS 13+, and Xcode Command Line Tools for compiling models.

v1.1.0 — BAAI/bge family support and a BERT backend

eeANE 1.1.0 adds support for the BAAI/bge model family: six newly verified models, a new BERT compile backend, and an opt-in for the pickle-based checkpoints some of these models are distributed as. Existing setups keep working as is — nothing changes unless you compile new models.

Six newly verified bge models

All of these compile with a single eeane compile <HF ID> and serve through the existing multi-model server, with accuracy self-checks against the FP32 originals passing on an M2 Mac (cosine ≥ 0.99998 for embeddings, sigmoid score difference ≤ 0.00001 for rerankers, Neural Engine placement 95.8–98.8%):

• BAAI/bge-m3 — multilingual embeddings, 1024 dimensions, 8192-token model (compile larger buckets with --buckets if you need more than the default 1024)
• BAAI/bge-reranker-base and BAAI/bge-reranker-large — rerankers; their 512-token maximum is handled automatically
• BAAI/bge-small-en-v1.5, bge-base-en-v1.5 and bge-large-en-v1.5 — English embeddings at three sizes (the small one answers a 128-token request in about 1.5 ms warm)

A third architecture family: BERT

The compiler now has a BERT backend for embedding models — the bge v1.5 series above is the verified example, and other BERT-based sentence-transformers checkpoints are expected to work the same way. BERT cross-encoder rerankers are deliberately refused with an explanation rather than compiled: the compiled graph would have to pin BERT's segment ids to zero, which silently changes what a query/document pair means to this architecture. ModernBERT and XLM-RoBERTa rerankers are unaffected.

Pickle-based checkpoints: --allow-pickle

Some bge models (bge-m3 among them) are distributed as pytorch_model.bin only, with no safetensors weights. eeANE keeps requiring safetensors by default — for both Hub downloads and local directories — and now offers an explicit opt-in:

eeane compile BAAI/bge-m3 --allow-pickle

The opt-in loads the checkpoint through torch.load(weights_only=True) and logs a warning. The new Checkpoint formats section of the README explains the trade-off honestly: weights_only=True reduces but does not eliminate the risks of the pickle format, so only use the flag with checkpoints from publishers you trust. When a repository has safetensors, the flag changes nothing and the .bin files are never downloaded.

Richer PyPI metadata

The PyPI page now carries keywords, trove classifiers, and project links, making the package easier to find and evaluate.

Requires Python 3.11 or 3.12, an Apple Silicon Mac on macOS 13+, and Xcode Command Line Tools for compiling models.

🤖 Generated with Claude Code

v1.0.0 — First stable release, now on PyPI

eeANE 1.0.0 is the first stable release. eeANE is now published on PyPI, the documentation has been rewritten around it, and a launchd guide covers running the server as a proper macOS service. There are no functional changes to the server or the compiler since 0.10.0 — an existing setup keeps working as is.

Install from PyPI

uv tool install "eeane[compile]"

is now the whole install story: no git URL, no tag to pin. pipx and plain pip + venv work the same way (pipx install --python python3.12 "eeane[compile]"), the server-only lightweight variant remains, and installing from the GitHub repository is still documented for development snapshots and pinned revisions. If you installed 0.10.0 from a git URL, reinstall from PyPI once (uv tool install --force "eeane[compile]") and future upgrades are just uv tool upgrade eeane. From this release on, every tagged release is published to PyPI automatically by CI via trusted publishing.

Run eeANE as a service

docs/launchd.md is a step-by-step guide to running eeane serve as a macOS user LaunchAgent — start at login, restart on crash — with a ready-made plist template in the same directory. The whole flow (bootstrap, health check, on-demand model load, restart, boot out) is verified on real hardware. On-demand loading makes an always-on agent nearly free while idle.

Documentation overhaul

Both READMEs (English and Japanese) were rewritten for 1.0: a highlights section, PyPI-first installation, a quick start, a performance section with measurement conditions stated, troubleshooting, known limitations, and a compact changelog table that links here instead of narrating every version. The known-limitations section now says explicitly what is and is not verified: all published measurements come from an M2 Mac, other Apple Silicon generations are expected to work but unverified — the self-check summary that eeane compile prints doubles as a compatibility report, and reports from other machines are very welcome as issues.

Acknowledgments

eeANE exists because Infinity showed how convenient a self-hosted, API-compatible embedding and reranking server can be, and eeANE's /rerank API deliberately follows Infinity's schema. The two projects complement each other: eeANE is Apple-Neural-Engine-only by design — for serving on Linux, Windows, NVIDIA/AMD GPUs or CPUs, or for a much wider model catalogue, use Infinity.

Requires Python 3.11 or 3.12 (uv resolves one automatically; pipx and pip + venv need you to provide it), an Apple Silicon Mac on macOS 13+, and Xcode Command Line Tools for compiling models.

🤖 Generated with Claude Code

v0.10.0 — Install from GitHub with one command

v0.10 turns eeANE into an installable Python package: one command installs it straight from this repository, eeane compile / eeane serve run as regular installed commands, and a crash affecting Python 3.12+ environments is fixed.

Install from GitHub

uv tool install "eeane[compile] @ git+https://github.com/xhighhongo41/eeANE@v0.10.0"

installs eeANE together with everything needed to both compile models and serve them — no clone, no manual environment. pipx and plain pip + venv work too, and a lightweight server-only variant keeps the installed environment down to eeANE's five runtime dependencies; the README's new Installation section covers all four. Every documented command is verified against a fully fresh environment: install → eeane compile → eeane serve → the complete verification suite passes with no reliance on any pre-existing cache.

The eeane command

The package now installs an eeane console command (serve, check-config, compile) and a top-level --version flag. python -m eeane and python -m eeane.server keep working as backward-compatible aliases, and the development workflow (clone + uv sync) is unchanged, documented in the README's new Development section.

Python 3.12+ crash fix

On CPython 3.12 and later, the server crashed a few seconds after its first prediction: Core ML releases the feature values wrapping a prediction's inputs from an internal dispatch queue that has no Python thread state, and on 3.12+ the final release of the backing numpy buffers from that thread takes the process down. The engine now keeps each compiled model's most recent inputs referenced and quarantines replaced or unloaded ones for a grace period, so the final release always happens on a Python thread holding the GIL. Verified live on Python 3.12 (server survives single and repeated requests; full verification suite passes) with no change in behavior on 3.11.

CI

A new install job builds the wheel and sdist, verifies the wheel ships only the eeane package, smoke-tests it in an isolated environment, and installs it with plain pip from both the checkout and a git URL.

Requires Python 3.11 or 3.12 (uv resolves one automatically; pipx and pip + venv need you to provide it), an Apple Silicon Mac on macOS 13+, and Xcode Command Line Tools for compiling models.

🤖 Generated with Claude Code

v0.9.0 — Queue hardening, robustness and batch-2 serving

v0.9 hardens the server under load: requests beyond the server's capacity now get clear, immediate answers instead of waiting forever, identical concurrent requests are computed once, shutdown drains what is in flight, a broken compute path fails loudly instead of returning poisoned numbers — and short-input embedding throughput can be raised ~25% with an opt-in batch-2 artifact.

Admission control and queue timeout

Inference requests pass admission control. [server] max_pending_requests (default 500, 0 = unlimited) caps how many are accepted at once — running and waiting combined — and the excess is rejected immediately with 429 and a Retry-After header (measured rejection time ~0.09 s even under a full queue; failed authentication never consumes a slot). A request that waits longer than [server] queue_timeout (default 600 s, 0 = disabled) for its turn is answered with 503 and Retry-After. A request whose inference has started is never interrupted: the timeout only bounds waiting.

[server]
# max_pending_requests = 500   # 0 = unlimited
# queue_timeout = 600          # seconds; 0 = no timeout

The defaults are sized well above real ingestion bursts: a document-ingestion style burst of 33 concurrent requests (448k tokens) completes all-200 with default settings.

Request coalescing

Identical concurrent requests — same model, same input — are served from a single computation ([server] coalesce_requests, default on). Eight identical requests fired at once completed in 2.75 s against 20.2 s with coalescing off (~7.3x) on a development Mac, with byte-identical responses. Only requests overlapping in time are merged; nothing is cached.

Graceful shutdown

On SIGTERM/Ctrl-C the server stops accepting connections and finishes every accepted request before exiting (measured under load: all 32 in-flight requests completed, clean exit). [server] graceful_shutdown_timeout bounds that wait; unset, the server waits as long as it takes.

Batch-2 serving (opt-in)

Compiling a second artifact with eeane compile <model> --buckets <S> --batch 2 lets the server predict two same-bucket inputs of one embedding request per Neural Engine call. A single 200-short-input request went from 5,792 to 7,234 tokens/s (+24.9%) on a development Mac, with per-text agreement between the batch-1 and batch-2 paths at cosine ≥ 0.9999999. Grouping never crosses requests and never reorders results; configurations without a batch-2 artifact behave exactly as before. Id-only [[models]] entries pick a compiled batch-2 artifact up from the cache automatically. The compiled-model record now keeps the batch-1 and batch-2 families separate, fixing a latent issue where a --batch 2 compile could overwrite the batch-1 record.

Non-finite output guard

A model output containing NaN or infinite values is now answered with a clear 500 error and an error log naming the model and bucket, instead of being passed on to poison a similarity search. This matters because eeANE requires the Apple Neural Engine: CPU-only execution of the compiled models was measured to produce non-finite outputs across all supported architectures, so this error usually means the ANE was not available to the process. The READMEs now state this under known limitations.

Read-ahead tokenization

A multi-input request tokenizes its next input on a worker thread while the current one is being predicted. Measured effect is modest (+1.6% on a concurrent burst, within noise for single requests — tokenization is small next to prediction); results, ordering and accuracy are unchanged.

Load-test client

tools/load_test.py builds deterministic, seeded request sets from the bundled corpus and fires them concurrently: an ingestion-style burst, a short-input sweep, an admission-limit probe (--expect-429) and a duplicate-request probe that checks responses are byte-identical.

Behavior changes

• Admission cap (500) and queue timeout (600 s) are active by default; previously the queue was unbounded. Set max_pending_requests = 0 and queue_timeout = 0 to restore the old behavior.
• New 429/503 responses carry Retry-After; clients that treat any non-200 as fatal will surface these as errors instead of hanging.
• No API or configuration compatibility is broken: every v0.8 config file remains valid.

Verification: 811 tests pass, CI green; default-config accuracy verification passes end to end with a 36-document rerank median of 5.612 s, level with v0.8.