[Feature] Add EmbeddingGemma 2 (sub-1B, multilingual, Apache 2.0) as a Smart Search model

[shrijayan]

• I have searched the existing feature requests, both open and closed, to make sure this is not a duplicate request.

The feature

Add google/embeddinggemma-2 as a selectable Smart Search model in Administration > Settings > Machine Learning > Smart Search.

Why this model is a strong fit

Google announced EmbeddingGemma 2 today. For the part Immich cares about, the text + vision config ({"audio_config": None}) is 440M parameters and maps images and search queries into one shared 768-d space — which is exactly the visual/textual pair split smart_search already assumes.

• 100+ languages, and it keeps EmbeddingGemma 1's multilingual text quality while fixing its weak code retrieval (+14% MTEB Code).
• Apache 2.0, so converted weights can be hosted under the immich-app HF org.

The edge case — this is the part I think matters most

Immich's smart search quality today has a cost floor. Looking at the searching.md tables, the multilingual options start at 3854–4675 MiB of model memory (ViT-SO400M-16-SigLIP2-384__webli, nllb-clip-base-siglip__v1), and the English-only default ViT-B-32__openai scores 69.9 recall. That means the good multilingual models are effectively locked to a decent server, and the ones that fit on small hardware are the weak ones.

EmbeddingGemma 2 is purpose-built to break that tradeoff:

Lever	What it gives	Practical effect on a small box
Modular loading	270M text-only / 440M text+vision / 740M full — unused encoders are never loaded, so you save both the weights and peak allocation	Smart Search on a NUC, mini PC, or Apple Silicon without a dedicated GPU
bf16 inference	Halves memory vs fp32, same 8-bit exponent as fp32	~2x headroom over an fp32 export on the same RAM
MRL truncation	768 → 512 / 256 / 128, re-normalized after slicing	1M vectors: ~1.5 GB at 768d vs ~250 MB at 128d (Google's numbers)
Sub-1B total	740M max across all four modalities	Small enough to stay resident and warm, so no reload penalty per query
Quantization already proven	Community mxfp4 / nvfp4 / qat builds exist	A 4-bit path is realistic rather than aspirational

Put plainly: this is one of the first strong multilingual retrieval models where "good search" and "runs on consumer hardware" stop being a trade-off. Google already ships LiteRT-LM, MLX, Ollama and LMStudio paths for it — the edge story is established, not theoretical.

I'm not claiming Immich users on small hardware are numerous, but right now the model tables effectively assume a server with headroom. If there were a 440M multilingual option that keeps recall near the 3.8 GB models, that changes who can self-host well.

Note float16 is a trap here — the model card is explicit that fp16 exceeds the activation range and returns NaN rather than raising, so it has to be handled in export, not left to users.

Audio is out of scope (Immich has no audio search). Video maps cleanly, since Immich already embeds a sampled frame through the vision encoder.

Why this is more than an allow-list entry

Contrast with fg-clip2-base #31529, which was two lines because it is open_clip-compatible. EmbeddingGemma 2 is not, so I think this needs a new ModelSource rather than an addition to _OPENCLIP_MODELS. Concretely:

Concern	What open_clip models do today	What EmbeddingGemma 2 needs
Tokenizer	CLIP BPE, loaded via tokenizer.json	Gemma SentencePiece, 262k vocab
Image preprocessing	preprocess_cfg.json, fixed center crop (224/256/384/…)	~280 vision tokens per image, resampler, no fixed crop
Query encoding	encodeText(query) verbatim	Needs task: search result | query: {query} prefix
Output dim	512 (occasionally 768)	768 native; CLIP_MODEL_INFO entry + smart_search migration
Numerics	fp16/fp32 both fine	bf16 or fp32 required
Weight layout	one model.onnx per role	separate vision / text graphs from one checkpoint

Two of these are behavioural rather than plumbing:

1. The task prompt is not optional. Omitting task: search result | query: ... on the query side does not error — it silently degrades ranking quality. That should be baked into the textual encoder, not exposed as an option.
2. bf16, not fp16. NaN-instead-of-exception needs enforcing in the export/session config.

What I can do

• immich-app/ml-models: a new embeddinggemma source + models.yaml entry
• machine-learning/immich_ml/models/constants.py: register the new source
• a GemmaEmbeddingVisualEncoder / GemmaEmbeddingTextualEncoder pair under models/
• export with torch.onnx.export in bf16, validated cos = 1.0 against the HF reference on all outputs
• publish converted weights to the immich-app HF org (the ML service pulls from immich-app/<model>)
• add the row to the searching.md benchmark tables

Three questions before I start:

1. Is this wanted at all? A new ModelSource is ongoing maintenance, and I don't yet have Immich-harness numbers to prove 440M beats the 3854 MiB incumbent. If the answer is "not worth it", I'd rather hear that than spend a weekend on an export nobody installs. Happy to run the benchmark first and come back.
2. What's the acceptance bar? Presumably a searching.md row with memory, execution time, and recall on Crossmodal-3600 / XTD-10 / Flickr30k, same harness as the existing numbers.
3. 768-d or truncate to 512? 768 means smart_search migrates for anyone who switches. MRL truncation to 512 keeps the current column, and Google's own numbers put 512d within ~0.2 MTEB of full 768d on text. I lean 512d for a first pass — and it's also the friendlier option for exactly the small-hardware case above — but that's a product call.

If a PR is easier to review than a feature request, I'm happy to open the export scaffolding against immich-app/ml-models first and let this thread be the decision on #1.

Platform

• Server
• Web
• Mobile

[github-actions[bot]]

This discussion has automatically been closed as it is likely a duplicate. We get a lot of duplicate threads each day, which is why we ask you in the template to confirm that you searched for duplicates before opening one. If you're sure this is not a duplicate, please leave a comment and we will reopen the thread if necessary.

[shrijayan]

Checked before opening, and this isn't a duplicate of any existing thread. Searches I ran on this repo's discussions (open and closed) for gemma, embeddinggemma, embedding model request, and in:title CLIP returned no EmbeddingGemma thread of any kind. EmbeddingGemma (v1 or v2) has never been requested here.

On the closest-looking thread, #31529 (fg-clip2-base): that one is genuinely a two-line change because it is open_clip-compatible — constants.py + CLIP_MODEL_INFO. EmbeddingGemma 2 is not open_clip-compatible, so it needs a new ModelSource. Specifically it needs a Gemma SentencePiece tokenizer (262k vocab) instead of CLIP BPE, ~280-token vision preprocessing instead of preprocess_cfg.json center-crop, a mandatory task: search result | query: ... prefix on the text side, 768-d output, and bf16 (fp16 returns NaN rather than raising). That's the substance of the request — it isn't a repeat of an existing ask for the same model.

If the detector flagged it against the general "add a CLIP model for Smart Search" cluster (#11845, #14958, #17135, #25762, #31529), those are all for text-image CLIP/SigLIP models with a 512-d CLIP-style interface. This one is a genuinely different architecture with a different integration path, and the reason I think it's worth the effort is the multilingual ones on that cluster are 3854–4675 MiB while this is 440M with MRL truncation.

Happy to reopen the conversation on any part of this if it's not the shape you'd want.

• Confirmed not a duplicate

[bo0tzz]

@shrijayan please read and understand https://github.com/immich-app/immich/blob/main/CONTRIBUTING.md#use-of-generative-ai

[shrijayan]

You're right, I missed that. I got excited when the model dropped and had an agent write this up, which is exactly what the guidelines ask people not to do. That's on me, and I'll write any future posts here myself.

Also thank you for actually running the benchmarks — that's way more effort than I expected anyone to put into this, and the numbers are useful regardless of what happens with the request. Happy to have the thread closed with a comment if that's the call, or left up for the benchmark tables if people find them useful. Either way I'm good with whatever you decide.

[bo0tzz]

It doesn't seem to me like you wrote this comment yourself either...

[Neo11Neo]

just wanted to write the exact same! Kudos for your suggestion!

[bo0tzz]

I've run some local benchmarks to compare this to our existing models. The OP claims that this model is good for "small hardware" don't hold up for me, and overall the gains (where they do exist) seem slim and the complexity of adding this model doesn't seem worth it to me - but final call is @mertalev's.

Setup: text→image retrieval recall (mean of R@1/5/10), fp32, CPU (i9-12900H).

• XTD-10: 1000 images, all 11 languages, every caption.
• Crossmodal-3600 (XM): full 3600-image gallery, 8 languages × 500 sampled captions.
• EmbeddingGemma 2 loaded text+vision only (no audio), query prompt task: search result | query: . "@N" = vision token budget (default 280).
• Incumbents via open_clip; ORT timings use the published immich-app ONNX visual models.

Model	XTD en	XTD mean	XM en	XM mean	ORT image	Torch image	Torch query	Torch RSS
ViT-B-32__openai	73.9	21.2	56.9	8.4	32 ms	48 ms	22 ms	1.6 GB
ViT-B-16-SigLIP2__webli	89.2	77.4	74.3	49.5	75 ms	109 ms	38 ms	3.3 GB
nllb-clip-base-siglip__v1	85.7	84.1	67.1	69.5	272 ms	299 ms	102 ms	5.2 GB
ViT-SO400M-16-SigLIP2-384__webli	90.1	81.9	73.9	55.2	1370 ms	1400 ms	165 ms	9.1 GB
nllb-clip-large-siglip__v1	88.4	87.1	70.1	73.7	–	1715 ms	312 ms	9.3 GB
EmbeddingGemma 2 @70	89.6	84.7	75.1	69.7	638 ms	624 ms	37 ms	3.0 GB
EmbeddingGemma 2 @140	90.4	85.8	76.9	70.9	–	1580 ms	50 ms	3.0 GB
EmbeddingGemma 2 @280	90.7	85.6	–	–	6134 ms	4106 ms	63 ms	3.0 GB

Crossmodal-3600 per language:

Model	en	zh	ja	ko	hi	ar	th	sw
ViT-B-16-SigLIP2__webli	74.3	64.9	58.7	70.3	25.3	58.5	34.5	9.8
ViT-SO400M-16-SigLIP2-384__webli	73.9	66.4	67.1	77.6	32.1	66.7	44.1	13.3
nllb-clip-base-siglip__v1	67.1	71.7	81.9	71.5	57.3	72.0	72.9	61.7
nllb-clip-large-siglip__v1	70.1	76.1	85.7	76.9	60.1	75.0	77.9	67.5
EmbeddingGemma 2 @70	75.1	81.8	89.6	81.6	50.2	71.1	82.5	25.9
EmbeddingGemma 2 @140	76.9	83.1	90.9	82.1	53.1	71.5	82.9	26.4

XTD-10 per language:

Model	en	zh	fr	de	it	ja	ko	pl	ru	es	tr
ViT-B-16-SigLIP2__webli	89.2	65.1	83.9	83.7	83.3	55.7	73.0	77.3	81.1	86.2	72.4
ViT-SO400M-16-SigLIP2-384__webli	90.1	69.2	86.7	86.4	86.4	62.8	79.5	84.5	86.1	88.4	80.9
nllb-clip-base-siglip__v1	85.7	82.5	84.9	84.7	85.5	82.9	82.2	83.7	84.1	84.9	84.4
nllb-clip-large-siglip__v1	88.4	85.3	88.1	87.2	87.9	86.0	85.7	87.4	87.4	87.2	87.3
EmbeddingGemma 2 @70	89.6	80.7	86.3	85.8	85.6	82.8	81.7	84.0	85.1	86.8	83.4
EmbeddingGemma 2 @140	90.4	82.5	86.9	86.7	87.1	83.7	82.5	85.1	85.9	87.4	84.9
EmbeddingGemma 2 @280	90.7	81.6	86.9	87.0	86.2	83.4	82.9	85.5	86.2	87.0	84.4

[mertalev]

It edges out the best SigLIP2 model for English and is significantly better for other languages, so it's on the Pareto frontier even if it is slower.

But I think the testing here doesn't tell the whole story since this model can use higher resolution inputs depending on token budget while the datasets here are probably low resolution images. This applies in the other direction as well since it can use a token budget of 70 or 140 instead of 280, which would be much faster.

Overall, I think it's a good addition if it can be exported to an efficient ONNX model like the rest of the catalog. It's architecturally very different, though, so would need some work compared to a traditional CLIP model.

Edit: It'd probably be good to test 512x512 models with a higher resolution dataset as well. 384 outperforming 512 in the eval table might just be because the images were all low res.