Repository navigation
Gemma 4 DualMode: generation + EmbeddingGemma-2-distilled retrieval from one shared backbone (E2B, HF + GGUF) #812
gtotaro86-ai
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Problem. Local RAG usually keeps two text models resident: Gemma 4 for generation and a separate embedding model (e.g. EmbeddingGemma 2) for retrieval. That is two backbones in memory and two runtimes.
Prototype. One Gemma 4 E2B-it backbone is used for both:
q_proj/o_proj, layers 15-33, rank 16), hidden state after layer 33, mean pooling, aLinear(1536 -> 768)head, L2 normalization (MRL 768/512/256/128).google/embeddinggemma-2, used only for distillation (about 12k pairs from 13 public datasets). It is not merged and not needed at runtime.k_proj/v_projLoRA in those layers has no effect; onlyq_proj/o_projmatter there.)Generation isolation. On the exact export, against a fresh unmodified
gemma-4-E2B-it: greedy token parity 6/6 prompts, max logit difference 0.0,embed -> generate -> embedandgenerate -> embed -> generatebitwise identical, 18 alternating switching patterns with 0 mismatches. In the GGUF runtime: 36 embed/generate switches in one process, 0 differences; generation is token- and logit-identical to the plain base GGUF.Retrieval (teacher EmbeddingGemma 2 on the same texts, 128 tokens, 500-doc BEIR/MTEB test subsets, never used for training or model selection):
Internal held-out validation: MRR 0.8585 (teacher 0.9203). We do not hide the gap: this is a research-grade retriever at roughly 71-79 % of the teacher's MRR, not index-compatible with EmbeddingGemma 2 vectors (cross-space retrieval is clearly worse than native). A first version trained on only 400 pairs scored SciFact MRR 0.038; the cause was embedding-space collapse, fixed by broader distillation.
Runtime / GGUF. HF/Safetensors (adapter + Python runtime) and a single GGUF file (Q5_K_M backbone + F16 head + F16 LoRA + metadata). Stock llama.cpp cannot load it: a small patch (368 lines, llama.cpp
f498f864, which already has EmbeddingGemma-2 support) adds a cache-free bidirectional embed graph forgemma4, and a small example holds one model with two contexts. The patch is a private-fork style patch, not an upstream proposal. GGUF Q5_K_M reaches 98.8 % (SciFact) / 94.8 % (NFCorpus) of the BF16 MRR; HF-vs-GGUF F16 mean cosine 0.9996.Memory (measured, not projected). PyTorch ROCm on an AMD Radeon 8060S UMA GPU (Phase 2 adapter, 3.4 MB smaller than the current one): 516 MB VRAM/UMA and 251 MB process RSS saved versus keeping a standalone EmbeddingGemma 2 text model next to Gemma 4. GGUF Q5_K_M on CPU (8 threads, Ryzen AI Max+ 392): file 3.64 GB, load 0.9 s (mmap), embed median 433 ms, generation 23.3 tok/s. No mobile or LiteRT numbers; we have not tested phones.
Links.
Question. Would the Gemma / AI Edge team be interested in a standardized embedding-mode interface for Gemma models, allowing a shared backbone to switch between causal generation and bidirectional retrieval? We would also welcome feedback on whether such a mode could fit LiteRT / AI Edge runtimes without requiring a second resident model.
Positioning: community research prototype, not an official Google release. In the public implementations we reviewed we did not find another Gemma 4 system that exposes both autoregressive generation and EmbeddingGemma-2-distilled dense retrieval from the same resident backbone; we do not claim to be the first.
All reactions