[Feature] Support FG-CLIP2-base (bilingual zh/en) as a Smart Search model

[TeemoSun]

I have searched the existing feature requests, both open and closed, to make sure this is not a duplicate request.

• Yes

The feature

What

Add fg-clip2-base (qihoo360/fg-clip2-base, 0.4B, bilingual zh/en, fine-grained) to the supported Smart Search models. The code change itself is two allow-list entries, already opened as PR #31528:

• machine-learning/immich_ml/models/constants.py: add "fg-clip2-base" to _OPENCLIP_MODELS
• server/src/constants.ts: add 'fg-clip2-base': { dimSize: 768 } to CLIP_MODEL_INFO

I verified this end-to-end locally: built the server and ML images from that branch, deployed a stock docker-compose stack, ran Smart Search over ~1.6k photos, and searched with both English and Chinese queries. Embeddings are 768-d and the smart_search column is migrated automatically via CLIP_MODEL_INFO.

The converted ONNX package (Immich cache layout, fp32, validated cos=1.0 against the torch reference) is already public: https://huggingface.co/pigzhou/fg-clip2-base-immich — I am happy to hand it over for upload to the immich-app HF org if accepted. The export script and the validation/benchmark harness are available on request.

Why (measured, same harness for all models, production preprocessing)

Compared against the strongest multilingual incumbent, nllb-clip-base-siglip__v1 (Immich's current best Chinese-capable model): recall@1/5/10 average, text→image, en/zh.

model	XM3600 en/zh	XTD-10 en/zh	overall
fg-clip2-base (768-d)	86.20 / 90.07	89.13 / 81.30	86.67
nllb-clip-base-siglip__v1	81.13 / 82.10	83.70 / 79.73	81.67

Protocol: XM3600 subsampled to 1000 (seed 1234), XTD-10 full 1000; identical ONNX packages and production preprocessing for every model. Absolute values are slightly inflated by the smaller XM3600 gallery; the comparison is within-harness (nllb/siglip384 English values land within ~1 point of the docs table). For context: the English-only default ViT-B-16-SigLIP-384__webli scores 80.20/84.60 on English but chance-level ~0.7% on Chinese.

FG-CLIP2-base wins every cell, raising both languages above the current multilingual ceiling, at the same 768-d embedding cost as nllb-clip-base-siglip__v1.

Conversion notes (for reproducibility)

• FG-CLIP2 inference uses no attention mask (official eval behavior); masking pads changes embeddings by cos ≈ 0.4–0.6 and hurts retrieval badly.
• Immich's fixed 512² center-crop vs native dynamic resolution: no recall difference on XM3600 (−0.28, noise level).
• Export: torch.onnx.trace (opset 17), batched text head, in-graph patchify, position interpolation burned as a constant; validated cos=1.0 on all outputs.
• Benchmarks: text 33–37 ms, image ~0.7–0.75 s per item, batch=1 on 8-core CPU f32.

Ask

• Would the team accept fg-clip2-base as a supported Smart Search model?
• If yes: could the ONNX be hosted under the immich-app HuggingFace org? The ML service downloads from immich-app/<model>. I can transfer my repo or provide the export script — whichever is easier to review.

Platform

• Server
• Web
• Mobile

[mertalev]

It seems fine to add it and others in the family to the catalog. The models would be exported and uploaded in ml-models. Note that the exports from this repo are slightly different from what exists on Hugging Face due to optimizations that haven't been rolled out yet.