feat(llm): switch default embedder nomic → EmbeddingGemma 300M#1952
Conversation
The current llama.cpp server can't load nomic-embed-text-v2-moe, which broke RAG, code-index, and agent-memory embedding on any up-to-date Lemonade build. Default to EmbeddingGemma 300M instead. Lemonade ships no built-in EmbeddingGemma, so it's registered as the custom user-model user.embeddinggemma-300m-GGUF (checkpoint ggml-org/embeddinggemma-300M-GGUF:Q8_0, recipe llamacpp) with the embedding=true pull flag — which sets the embeddings label explicitly and avoids the #1745 auto-label bug. It's 768-dim, same as nomic, so FAISS index shapes and EMBEDDING_DIM stay unchanged. Memory vectors are model-specific and the dim check can't catch a model swap (both 768-dim), so memory now records the embedder name and clears+re-embeds stale vectors when it changes. RAG/code-index caches already self-invalidate on embedding_model change.
github-actions
Bot
added
documentation
|
Verdict: Approve — safe to merge as-is; the notes below are optional polish, not blockers. This swaps GAIA's default embedder from The bottom line reviewers should know: this only registers the pull shape — the live Lemonade contract (fresh cold-pull on a real box) still needs the boundary test the author left unchecked in the test plan. Nothing in the diff can prove that end-to-end, which is exactly the right call to flag. No security concerns. 🔍 Technical details🟢 Minor / nits1. 2. Procedures embeddings are cleared but not re-backfilled on the migration run (src/gaia/agents/base/memory.py:445) 3. EMR disk-space note (docs/guides/emr.mdx) — the per-model row dropped from 522 MB → ~334 MB but the "at least 25 GB free" guidance is unchanged. Fine (headroom), just noting the two lines were reviewed together. Strengths
|
Validated against a live Lemonade Server (10.9.0): EmbeddingGemma is a custom user-model, so /v1/pull registers it as user.embeddinggemma-300m-GGUF but /v1/models lists it under the stripped id embeddinggemma-300m-GGUF (/load and /embeddings accept either form). The availability check in ensure_model_downloaded compared raw strings, so it never matched the registered model and re-pulled on every call — an observed hang. Add a user.-prefix-tolerant name matcher and use it in the availability, download-wait, and loaded-model checks. Runtime validation now passes: ensure_model_downloaded returns immediately, gaia init resolves the embedder, and a full RAG index→retrieve→answer round-trip works. Also refines the embedder docs (agent-ui/code-index/rag) to match actual cache-invalidation behavior.
Generalize the embedder hardware test to validate EmbeddingGemma 300M on both backends GAIA uses: the GGUF/llamacpp variant (user.embeddinggemma-300m-GGUF, the non-NPU default replacing nomic, #1952) and the FLM/NPU variant (embed-gemma-300m-FLM, #1744). The vector-level checks — 768-dim, finite, non-zero, deterministic, distinct-texts-differ — are shared and parametrized across both backends; the chat+embedder co-residency check stays FLM-specific. Each backend gates on the live Lemonade catalog and skips when absent, so the GGUF variant runs on GPU/Vulkan boxes and the stx CI runner while FLM runs on Ryzen AI NPU runners — neither produces a false failure where its model isn't present. Verified locally: GGUF variant passes against a real EmbeddingGemma GGUF model (4 passed), FLM variant skips cleanly with no server error.
|
🟡
🔍 Technical details
health = self._llm_client.health_check()
running = [m.get("id", "") for m in health.get("all_models_loaded", [])]
if self.config.embedding_model not in running: # ← raw string compare
self._llm_client.load_model(
self.config.embedding_model,
llamacpp_args="--ubatch-size 2048 --split-mode none",
)Fix — swap the membership test for from gaia.llm.lemonade_client import _model_ids_match
if not any(_model_ids_match(rid, self.config.embedding_model) for rid in running):
self._llm_client.load_model(...)Secondary: |
|
🟡 The 🔍 Technical details
if not self.llm_client.ensure_model_downloaded(
self.config.embedding_model,
checkpoint=mr.checkpoint,
recipe=mr.recipe,
embedding=mr.embedding,
):
raise RuntimeError(
f"Failed to register/download embedding model ..."
)
if mr and self.config.embedding_model.startswith("user."):
self._llm_client.ensure_model_downloaded( # ← return value ignored
self.config.embedding_model,
checkpoint=mr.checkpoint,
recipe=mr.recipe,
embedding=mr.embedding,
)Fix: mirror the RAG pattern — check the bool return and raise a |
…ingGemma Two fixes for the embedder switch: 1. _download_models passed checkpoint/recipe/embedding=None to ensure_model_downloaded for EVERY model, including built-ins — which broke test_npu_profile_pulls_builtin_model_without_recipe and risked Lemonade treating a built-in pull as a custom registration (#1655). Only user.-namespaced models now carry those kwargs; built-ins are pulled by name. 2. EmbeddingGemma is validated on Lemonade v10.9.0; older bundled llama.cpp builds fail to load it. Bump LEMONADE_VERSION to 10.9.0 and floor the chat/rag/all init profiles at 10.9.0 so init fails loudly instead of the embedder failing at first RAG index. reset-lemonade.ps1 fallback bumped to match.
|
🟡 The The health endpoint returns the model's stripped id ( 🔍 Technical details
running = [m.get("id", "") for m in health.get("all_models_loaded", [])]
if self.config.embedding_model not in running: # <-- plain `in`, no prefix strip
self._llm_client.load_model(...)The from gaia.llm.lemonade_client import _model_ids_match
...
if not any(_model_ids_match(r, self.config.embedding_model) for r in running):
self._llm_client.load_model(...) |
The LEMONADE_VERSION bump to 10.9.0 left hardcoded 10.8.1 references in docs/cpp/setup.mdx, docs/guides/npu.mdx, and cpp/README.md, which util/check_doc_versions.py flags as mismatches (fails the validate + code-quality jobs). Sync them to 10.9.0.
…mbedder # Conflicts: # src/gaia/agents/base/memory.py # src/gaia/agents/base/memory_store.py
|
🔴
DEFAULT_EMBEDDING_MODEL = "nomic-embed-text-v2-moe-GGUF"The
🔍 Technical detailsRoot cause: Files to fix:
# before
DEFAULT_EMBEDDING_MODEL = "nomic-embed-text-v2-moe-GGUF"
# after
from gaia.llm.lemonade_client import DEFAULT_EMBEDDING_MODEL # "user.embeddinggemma-300m-GGUF"
# before
assert get_embedding_model_for_device(device) == "nomic-embed-text-v2-moe-GGUF"
# after (import EMBEDDING_MODEL from registry or lemonade_client)
assert get_embedding_model_for_device(device) == "user.embeddinggemma-300m-GGUF" |
…mbedder # Conflicts: # .github/workflows/test_embeddings.yml # .github/workflows/test_rag.yml
#1761 shipped the NPU FLM hardware test as tests/test_gemma_embedder_hw.py but its workflow (test_npu_embedder.yml) and the file's own docstring both reference tests/test_npu_flm_embedder_hw.py — so the job failed with 'file or directory not found'. Rename the file to the intended name.
|
🟡 🔍 Technical details
if mr and self.config.embedding_model.startswith("user."):
self._llm_client.ensure_model_downloaded( # return value discarded
self.config.embedding_model,
checkpoint=mr.checkpoint,
recipe=mr.recipe,
embedding=mr.embedding,
)Compare to if mr and self.config.embedding_model.startswith("user."):
if not self.llm_client.ensure_model_downloaded(...):
raise RuntimeError(
f"Failed to register/download embedding model ..."
)Fix: mirror the |
Two failures introduced on main by the #1748 (embedding cache) × #1761 (per-instance embedder) interaction and #1371 (schedule), which this branch inherits via the merge: 1. MemoryMixin._embed_text/_get_embedder referenced self._embedding_model directly, which is only set in init_memory — so the embedding-cache unit tests (bare mixin) failed with AttributeError, and the cache key used the module default while the embed call used the instance model (mismatched for the NPU FLM embedder). Resolve the active model/dim via helpers that fall back to the module default, and key the cache by the same resolved model. 2. schedule/daemon.py reimported datetime inside next_fire_time (W0404) — already imported module-level. Removed.
|
🟡 The 🔍 Technical details
if not self.llm_client.ensure_model_downloaded(
self.config.embedding_model,
checkpoint=mr.checkpoint,
recipe=mr.recipe,
embedding=mr.embedding,
):
raise RuntimeError(
f"Failed to register/download embedding model ..."
)
if mr and self.config.embedding_model.startswith("user."):
self._llm_client.ensure_model_downloaded( # ← return value dropped
self.config.embedding_model,
checkpoint=mr.checkpoint,
recipe=mr.recipe,
embedding=mr.embedding,
)Fix: mirror |
Root cause of the "Server failed to start" infra failures in the api, embeddings, and rag workflows: they launched `lemonade-server serve` inside a Start-Job, but the `lemonade-server` shim was removed in Lemonade v10.5 — on 10.9.0 it no longer exists, so the command failed instantly, the 60s health poll timed out, and the finally block discarded the server output (zero diagnostics). Fix: route all three through the shared start-lemonade.ps1 launcher (already used by the 5 healthy Lemonade workflows), which launches LemonadeServer.exe with a bare --port, captures stdout/stderr to log files, waits for health, and dumps those logs on failure. Restructure each into start → test → always-cleanup steps so the detached server's LEMONADE_PROCESS_ID (exported via GITHUB_ENV) is visible to the test and teardown steps, and surface the server logs whenever model setup/verify fails. Adds a -NoModel switch to start-lemonade.ps1 (start server only) so callers that register a custom user-model themselves (the EmbeddingGemma embedder via LemonadeClient) can reuse the robust launcher, and dumps server logs on a startup timeout.
|
🟡 🔍 Technical details
if mr and self.config.embedding_model.startswith("user."):
self._llm_client.ensure_model_downloaded( # return value discarded
self.config.embedding_model,
checkpoint=mr.checkpoint,
recipe=mr.recipe,
embedding=mr.embedding,
)When that returns if mr and self.config.embedding_model.startswith("user."):
if not self._llm_client.ensure_model_downloaded(
self.config.embedding_model,
checkpoint=mr.checkpoint,
recipe=mr.recipe,
embedding=mr.embedding,
):
raise RuntimeError(
f"Failed to register/download embedding model "
f"{self.config.embedding_model!r}. Ensure Lemonade Server "
f"is running, then retry (or run `gaia init --profile rag`)."
) |
…run)
Follow-up to the launcher fix. The multi-step split (start server in one
step, test in the next) failed because GitHub Actions reaps a step's child
processes at step end — the detached LemonadeServer.exe was gone by the test
step ("connection refused" on :13305). The healthy workflows (examples, npu)
run start-lemonade.ps1 + tests in a SINGLE step for exactly this reason.
Collapse the api/embeddings/rag jobs back to one step each: start-lemonade.ps1
(-NoModel) then register/verify + pytest in the same shell, with a finally
that stops the server. start-lemonade.ps1 now also sets $env:LEMONADE_PROCESS_ID
on the live process env (not just GITHUB_ENV, which only reaches later steps)
so same-step cleanup works.
The prior push already proved the launcher itself works: CI logs showed
"✅ LEMONADE SERVER READY" in ~2s where the old `lemonade-server serve` shim
timed out.
|
🔴 🟡 Same function also silently drops a 🔍 Technical details🔴 running = [m.get("id", "") for m in health.get("all_models_loaded", [])]
if self.config.embedding_model not in running:
from gaia.llm.lemonade_client import MODELS, LemonadeClient, _model_ids_match
...
if not any(_model_ids_match(r, self.config.embedding_model) for r in running):🟡 Silent failure on download — if mr and self.config.embedding_model.startswith("user."):
self._llm_client.ensure_model_downloaded(...) # return value discarded
|
The RAG job reached the embeddings round-trip (server up, models pulled, embedder registered+loaded) but failed with a raw KeyError: 'data' — a freshly loaded embedder can briefly return a non-standard body. Retry the embeddings call up to 5x and, on persistent failure, print the actual response instead of a KeyError so the fault is diagnosable.
|
🟡 The "is it already running?" check in the code-index embedder init breaks with the new 🔍 Technical details
running = [m.get("id", "") for m in health.get("all_models_loaded", [])]
if self.config.embedding_model not in running:
self._llm_client.load_model(
self.config.embedding_model,
llamacpp_args="--ubatch-size 2048 --split-mode none",
)With The fix — import and apply from gaia.llm.lemonade_client import MODELS, LemonadeClient, _model_ids_match
...
if not any(_model_ids_match(m, self.config.embedding_model) for m in running):
self._llm_client.load_model(...)Note: |
… sticks Root cause of the RAG/embeddings 501 'This server does not support embeddings. Start it with --embeddings': a shared runner had embeddinggemma-300m-GGUF already registered WITHOUT the embeddings label (stale registration from a prior job / #1745 auto-label bug), so ensure_model_downloaded short-circuited on 'already downloaded' and never applied the label — llama-server then loaded the model without --embeddings. For custom embedding models, delete any existing registration first so the subsequent pull re-registers with embedding=True and the label is guaranteed. Verified locally: delete + re-register + /v1/embeddings returns dim=768.
2 participants
Why this matters
The current llama.cpp server can't load
nomic-embed-text-v2-moe— GAIA's default embedder — so on any up-to-date Lemonade build, RAG, code-index, and agent-memory embedding silently break (the model fails to load; retrieval and memory stop working). This switches the default to EmbeddingGemma 300M, which loads cleanly on current llama.cpp. It's 768-dim (same as nomic), so index shapes and retrieval behavior are unchanged; existing indexes/caches re-embed automatically on next run.EmbeddingGemma is a Lemonade custom user-model:
/v1/pullregisters it asuser.embeddinggemma-300m-GGUF(checkpointggml-org/embeddinggemma-300M-GGUF:Q8_0,recipe=llamacpp,embedding=true— the flag sets theembeddingslabel explicitly, sidestepping the #1745 auto-label bug). Note that Lemonade then lists it under the stripped idembeddinggemma-300m-GGUFwhile/loadand/embeddingsaccept either form — so the availability check isuser.-prefix-tolerant to avoid a re-pull loop.Live validation (Lemonade Server 10.9.0)
ensure_model_downloaded("user.embeddinggemma-300m-GGUF")resolves the registered model instantly (no re-pull hang)gaia init --profile chatdownload path resolves both models cleanly/embeddingsreturns 768-dim vectors under theuser.-prefixed nametest_lemonade_client,test_memory_store,test_code_index,test_tool_call_priority,test_llamacpp_backend,test_rag), incl. new pull-shape + namespace-matcher tests;black/isortcleanStill to verify (couldn't run in this env)
/pullregister-from-scratch wasn't exercised end-to-end.anthropicSDK (not installed) and a realANTHROPIC_API_KEY(currently a dummy value); rungaia eval agent --category rag_quality --agent-type docand--comparetotests/fixtures/eval_baselines/gemma-4-e4b-d71cd914/scorecard_rag_quality.jsonin an env with judge access.