feat(agents): consolidate every agent onto Gemma-4-E4B at one context size#2284
Conversation
github-actions
Bot
added
documentation
Verdict: Request changes (small, mostly docs — the code is solid)This puts every agent on
One thing to confirm, not necessarily block: several agents that were deliberately lightweight (MCP bridge 4096, SD/VLM 8192, minimal 4096) now request 65536 on GPU. That's inherent to the consolidation, but it raises the memory floor 8–16× for those paths and isn't called out in the description — worth a sentence confirming it's intended. Nits (docs/comments) are in the details below. 🔍 Technical details🟡 ImportantEMR guide contradicts the EMR code change — Eval gate pending on an LLM-affecting change — base agent default flips from Qwen 35B to Gemma-4-E4B ( Confirm the lost small windows are intended — 🟢 MinorSame model, two download sizes — Stale "larger/coding model" comments — now that the id resolves to a 4B general model, these comments mislead: CI runner label routing never fires — Pattern note (not itemized): a broader sweep of
Strengths
|
bbd102a to
0470fad
Compare
|
🟡 One doc update was missed that directly contradicts the new GPU context-size behavior.
🔍 Technical details
The diff updated the
The instruction should read
|
|
🟡 Two new issues worth fixing before merge. 1. Doc mismatch on download size — 2. Preflight ctx check still uses the old 32K threshold — 🔍 Technical detailsDoc size inconsistency:
Preflight threshold mismatch:
|
de66e63 to
e4b4b1d
Compare
… size Switching agents evicted and cold-reloaded the model. Agents that left model_id unset fell back to Qwen3.5-35B-A3B-GGUF while chat and email used Gemma-4-E4B, so moving between them forced a full unload/reload — measured at 7.9s on a warm local Lemonade. Every agent now shares DEFAULT_MODEL_NAME, so the resident model is reused instead: three consecutive agent switches now cost 0.00s each with the llama-server pid unchanged. Context size follows the device profile rather than the agent. GPU_CTX_SIZE (65536) and NPU_CTX_SIZE (32768) replace the per-agent literals that ranged from 4096 to 65536; profile_ctx_size() keys on device because the NPU's FLM build cannot load above 32768 and would fail outright at the GPU window. A machine runs one profile, so exactly one (model, ctx_size) pair is resident. Measured no latency cost from the larger window: median 1.4s at 65536 vs 1.5s at 32768 on an identical 1631-token prompt. Out of scope and unchanged: the Summarizer's Qwen3-4B-Instruct-2507-GGUF, SD's SDXL-Turbo, the voice models, and Qwen3-VL-4B. The two legacy Qwen entries in the MODELS registry stay so existing pinned sessions keep resolving.
Adds the regression guards for the consolidation's two load-bearing
assumptions, both verified against a live Lemonade 10.10.0.
Residency (tests/integration/test_lemonade_model_residency.py): /health reports
max_models PER TYPE ({"llm": 1, "embedding": 1, ...}), so one LLM and one
embedder are already co-resident at default config — GAIA sets
max_loaded_models nowhere, and setting it to 2 would license a second LLM
rather than the pair. Observed directly: swapping the LLM changed its pid while
the embedder held the same pid throughout, so RAGSDK's scoped-unload assumption
(#1544) still holds. The swap test asserts the LLM pid actually turned over
before checking the embedder, because an earlier version passed in 0.32s
without triggering any load at all — _ensure_model_loaded early-returns when
the resident ctx already satisfies GAIA, so a same-model reload proves nothing.
Context: profile_ctx_size() is covered for gpu/cpu/npu, case and whitespace
normalisation, and the unknown-device default. The NPU case is the reason the
constants are not collapsed into one number.
`gaia eval agent` needs a real Lemonade on AMD hardware, so this runs on the Windows `stx` pool — PowerShell throughout, Python via setup-venv and Lemonade via install-lemonade, per the lab-runner convention. Running it on a dev machine is not a substitute: a 4B model elsewhere takes ~7 min/scenario and the committed baselines were recorded on AMD, so only this pool yields comparable numbers. Reports by default rather than failing. --compare exits 2 on any regression, and some regressions here are expected and accepted (Analyst and the tool_selection set dropped from a 35B MoE to a ~4B dense model), so a hard gate would sit permanently red and gate nothing. `enforce: true` makes it blocking once accepted regressions are folded into refreshed baselines. The paths filter covers every LLM-affecting surface, not just the files this change happened to touch — rag/sdk.py carries the RAG default model and cli.py resolves the context size, and a narrower filter would silently skip the gate on exactly the PR that needed it. Installs the chat and analyst hub agents explicitly and asserts `doc` and `data` resolve before evaluating: without them every scenario fails to find an agent and the run reports a catastrophic regression that never happened.
Documents the new single-model default and the per-profile context constants, and corrects the download sizes that referenced the retired 35B model. Sizes are measured from a real Lemonade install, not estimated: the Gemma download is 5.56 GB (4.64 GB Q4_K_M weights plus a 0.92 GB vision projector), which also corrects the Docker guide's ctx-size guidance — 8192 is below the window every agent now requests.
Both branches added the identical path, so whichever merged second would hard-conflict. #2283 is the CI-only PR that owns this workflow and is being reworked to a weekly schedule; keeping a copy here would overwrite that. This PR now depends on #2283 landing first, then dispatches that workflow as its merge gate.
#2244 gave BuilderAgent a preference list resolved against what is actually installed, which is better than a hardcoded default — but it listed Qwen3.5-35B-A3B-GGUF first. On a machine that still has the 35B installed the builder would pick it, evicting and cold-reloading the model every other agent shares, which is exactly the churn this branch removes. Gemma now leads; the 35B stays last so an existing install keeps working. Also points the context-overflow probe's threshold at GPU_CTX_SIZE instead of repeating 65536, so it tracks the profile window rather than drifting from it.
e4b4b1d to
aae4b4f
Compare
|
🟡 Fix the Jira guide to match the Docker guide: "about 5.6 GB (a 4.6 GB Q4_K_M weights file plus a 0.9 GB vision projector)". 🔍 Technical details
Both describe |
2 participants
Switching agents used to evict and cold-reload the model. Agents that left
model_idunset fell back toQwen3.5-35B-A3B-GGUFwhile chat and email usedGemma-4-E4B-it-GGUF, so moving between them forced a full unload/reload — 7.9s measured on a warm Lemonade. Every agent now shares one model id and one context window per device profile, so the resident model is reused: three consecutive agent switches cost 0.00s each with the llama-server pid unchanged.Important
Depends on #2283, and the eval gate is not green yet. The eval workflow now lives in #2283 (this branch previously added the same path — that copy was dropped to avoid a hard conflict). #2283 must merge first, then dispatch
Agent Eval — Gemma-4-E4B consolidationagainst this branch as the merge gate. CLAUDE.md requires that run for LLM-affecting changes. Partial results so far:budget_querybaseline 9.95 → 9.90 (flat). No full category has completed.What was checked empirically, not assumed
max_loaded_modelswas going to be added to keep the LLM and embedder co-resident. Measuring/api/v1/healthon Lemonade 10.10.0 showed it is unnecessary:Capacity is accounted per model type, so an LLM and an embedder are already co-resident at default config; setting the knob to 2 would license a second LLM, not the pair. Forcing an LLM swap confirmed it — the LLM pid turned over while the embedder held the same pid throughout. So the knob was not added, and the behaviour is pinned by tests instead.
Related: Lemonade runs one
llama-serversubprocess per model on its own port (8001/8002 observed), so there is no single global "slot" to raise.Notes for review
rag_qualitywill not mean "no impact." Most of its scenarios run thedocagent, which comes fromhub/agents/python/chatand was already on Gemma. The scenarios that actually exercise this change arecsv_analysis(thedata/Analyst agent, previously Qwen 35B) and thetool_selectionset. Regressions there are the accepted 35B→4B tradeoff.GPU_CTX_SIZE(65536) andNPU_CTX_SIZE(32768) — the NPU's FLM build cannot load above 32768, so collapsing them would either break NPU or cap GPU doc-Q&A at 32K and reopen [Bug]: 1.4MB file indexed no response to queries in the UI #1030. Measured no latency cost from the larger window: median 1.4s @65536 vs 1.5s @32768 on an identical 1631-token prompt.embed-gemma-300m-FLMso chat and embeddings stay on the same backend.Qwen3-4B-Instruct-2507-GGUF, SD'sSDXL-Turbo, voice models,Qwen3-VL-4B. The two legacy Qwen entries in theMODELSregistry stay so existing pinned sessions keep resolving.Two findings from the first runner execution
Worth carrying into #2283, since they are properties of the eval, not of one workflow file:
chatandanalysthub agents installed. Scenarios select their agent viaagent_type— 15 requestdoc, one requestsdata.doc/chat/filecome from the chat package's entry points, not fromdocqa(which registers asdocqaand no scenario requests). Without them every scenario fails to resolve an agent and the run reports a catastrophic regression that never happened. A pre-flight assertion thatdocanddataare registered turns that into a 60-second loud failure.run:blocks — em dashes producedThe string is missing the terminator. Keep inline PowerShell ASCII-only.Follow-up (not in this PR)
nomic-embed0.48 GB), correcting stale~3 GB/18.6 GB/25 GBclaims across the EMR guide, profile tables and hardware-advisor playbook. That PR is now closed, so this work is stranded on a closed branch and should be re-landed.device.py's_MIN_GPU_VRAM_GB = 24.0is deliberately untouched. Its rationale — "24 GB to fit Qwen3.5-35B (~17 GB)" — is now obsolete: Gemma-4-E4B measures 5.56 GB, andLITE_MIN_MEMORY_GB = 5.0is precedent for a lower bar. Lowering it changes the hardware support matrix, which is a product call rather than a side effect of a model swap.Test plan
Agent Eval — Gemma-4-E4B consolidationworkflow against this branch. This is the merge gate.docanddataregistered before any scenario executes.csv_analysisandtool_selectionfor the expected 35B→4B regression; decide whether to refresh baselines with--save-baselineand re-dispatch withenforce: true.python -m pytest tests/unit/ -q— 6786 passed post-rebase; 5 failures are pre-existing on a cleanmain(test_file_write_guardrails, 3xtest_init_command,test_memory_router).python util/lint.py --all— passes.python -m pytest tests/integration/test_lemonade_model_residency.py -v— 4 passed against live Lemonade 10.10.0.