CRITICAL: Hermes 64K context floor overrides VRAM-fit context -> llama-server OOM-killed on low-VRAM GPUs

[patil2001]

CRITICAL: Hermes 64K context floor overrides VRAM-fit context → llama-server OOM-killed on low-VRAM GPUs

Summary

When Hermes is enabled on a low-VRAM GPU (Tier 0/1, e.g. GTX 1650 with 4GB), the installer's "Hermes 64K context floor" unconditionally raises MAX_CONTEXT to 65536, discarding the VRAM-bounded context that the catalog model selector (scripts/select-model.py) computed. The resulting --ctx-size 65536 makes llama-server exhaust all 4GB of VRAM and the container is SIGKILLed (exit 137), taking the entire local LLM stack offline.

Reproduction (verified on a real GTX 1650, 4GB, 7GB RAM, Windows installer v2.5.3, Tier 0)

1. Install with Hermes enabled. Phase 02 selector computes MAX_CONTEXT=8192 (VRAM fit), MODEL_RECOMMENDED_ALTERNATIVES=qwen3.5-2b-q4:8192:3, reason: "gives 8K context".
2. Phase 03 (installers/phases/03-features.sh:92-101) sees Hermes on, MAX_CONTEXT < 65536, and clobbers it: MAX_CONTEXT="$HERMES_CONTEXT_SIZE" → 65536.
3. .env ends with CTX_SIZE=65536; docker-compose.base.yml:43 passes --ctx-size 65536 to llama-server.
4. llama.cpp log: common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 999, abort; VRAM breakdown reaches 4095 = 788 + (2487...) + 819 = full; container exits 137 (SIGKILL).

Same logic exists in all three installers:

• installers/phases/03-features.sh:92-101
• installers/windows/phases/03-features.ps1:152-161
• installers/macos/install-macos.sh:1524-1529

Impact

• Every Tier 0/1 install with Hermes enabled (Hermes is the default agent and always-on for local mode) ships with a dead llama-server: chat, Open WebUI, Hermes, Perplexica, and ODS Talk all fail.
• Fixing requires manually lowering CTX_SIZE in .env after install (which is what unblocks it: at CTX_SIZE=8192 the same server is healthy at 1889/4096 MiB VRAM, ~84 tok/s).

Why it's a bug, not "by design"

The codebase already models VRAM fit: select-model.py picks the largest model+context that fits and its test (tests/test-tier-map.sh) asserts the selector emits a VRAM-ceiling note. The Hermes floor is supposed to raise context only where hardware allows — the comment in tests/test-hermes-context-floor.sh even says it "must not inflate 8GB-class installs to 128K and starve llama-server VRAM" — but the implementation has no VRAM gate at all, so it starves even 4GB cards down to zero.

Proposed fix (single scope)

Gate the Hermes floor on VRAM capacity in all three installers:

• Discrete NVIDIA/AMD GPU with GPU_VRAM < 8192 MiB → keep the VRAM-fit MAX_CONTEXT, warn that Hermes needs 64K but the GPU cannot hold it, and record the reason.
• CPU backend (VRAM-bounded by RAM) and macOS unified-memory → keep current floor behavior.
• Respect an existing explicit HERMES_CONTEXT_SIZE override.

This preserves Hermes-on-64K for capable hardware while stopping the guaranteed OOM crash on 4GB/6GB cards.

Affected files

• installers/phases/03-features.sh
• installers/windows/phases/03-features.ps1
• installers/macos/install-macos.sh
• tests/test-hermes-context-floor.sh (add low-VRAM regression case)