[Bug]: LM Studio probe returns None when model not currently loaded — ignores max_context_length fallback

[nachilau]

Bug Description

_query_local_context_length() in agent/model_metadata.py queries LM Studio's native /api/v1/models endpoint and reads loaded_instances[0].config.context_length, but only when the model is currently loaded. When the model isn't loaded (e.g. JIT enabled, fresh server start, model just unloaded by TTL), loaded_instances is [] and the probe returns None — even though the same response always carries max_context_length (the model's training-time max), which would be a usable fallback.

The probe then falls through to the hardcoded family default (e.g. 256K for gemma-4, 64K for unknowns). For a user who actually loads the model with a smaller context window than the family default (very common — LM Studio's loader UI defaults to 4K–8K depending on free VRAM), this lets Hermes treat the model as having far more context than its KV cache can hold. The conversation grows past the loaded window before any compression triggers, and on the next turn LM Studio either silently truncates or — under VRAM pressure — the GPU driver fails to allocate KV cache buffers and the system hangs/BSODs.

This is distinct from #47200 (remote LM Studio is is_local_endpoint()-gated), #25989 (manual preload uses MINIMUM_CONTEXT_LENGTH), and #30178 (per-model override regression). Those affect different code paths; this one is in the probe itself.

Steps to Reproduce

1. Start LM Studio locally on port 1234 with JIT enabled. Do not load any model into memory yet.
2. Configure Hermes with an LM Studio custom_provider but without any per-model context_length override:

   custom_providers:
     - name: LM Studio
       base_url: http://localhost:1234/v1
       api_key: lm-studio
       discover_models: false
       model: google/gemma-4-12b-qat
       models:
         - google/gemma-4-12b-qat

3. Run a one-shot resolver call against the unloaded model:

   from agent.model_metadata import get_model_context_length
   ctx = get_model_context_length(
       model="google/gemma-4-12b-qat",
       base_url="http://localhost:1234/v1",
       config_context_length=None,
       provider="lmstudio",
       custom_providers=[...],  # as above
   )
   print(ctx)

4. Observe Hermes' INFO log lines:

   Could not detect context length for model 'google/gemma-4-12b-qat' at http://localhost:1234/v1 — defaulting to 256,000 tokens (probe-down). Set model.context_length in config.yaml to override.
   Using hardcoded context length 256,000 for model 'google/gemma-4-12b-qat' (custom endpoint, catalog match on 'gemma-4')
   

5. Manually inspect what LM Studio actually returns:

   curl -s http://localhost:1234/api/v1/models | jq '.models[] | select(.key=="google/gemma-4-12b-qat")'

   Output (verified locally on LM Studio 0.3.x, model unloaded):

   {
     "key": "google/gemma-4-12b-qat",
     "loaded_instances": [],
     "max_context_length": 262144,
     ...
   }

Expected Behavior

When loaded_instances is empty, the probe should fall back to max_context_length from the same response. That gives the training-time max (262,144 in this example), which is still better than ignoring a known-good number and resolving via a coarser fallback.

Optionally and even better: when loaded_instances is non-empty, prefer loaded_instances[0].config.context_length (current behaviour); when it's empty, return max_context_length. Either way, never return None if the model is listed and max_context_length is present.

Actual Behavior

_query_local_context_length returns None whenever loaded_instances is [], even though max_context_length is in the same JSON response. The resolver falls through to family-default heuristics (256K for gemma-4, etc.), which can be much larger than what the user will actually load the model with at JIT time. Hermes then treats the conversation budget as far larger than the actual loaded KV cache, and inputs sized to the wrong budget can cause LM Studio truncation or — when VRAM is already tight — drive nvlddmkm into STATUS_INSUFFICIENT_RESOURCES (VIDEO_TDR_FAILURE 0x116) on NVIDIA hardware.

Affected Component

Configuration (config.yaml, .env, hermes setup); Agent Core (conversation loop, context compression, memory)

Operating System

Windows 11 (RTX 3090, 24 GB VRAM)

Hermes Version

(version stamp will be filled in by the issue submission flow)

Root Cause Analysis

Probe code in agent/model_metadata.py at _query_local_context_length() (around lines 1337–1349 on current main):

if server_type == "lm-studio":
    resp = client.get(f"{server_url}/api/v1/models")
    if resp.status_code == 200:
        data = resp.json()
        for m in data.get("models", []):
            if _model_id_matches(m.get("key", ""), model) or _model_id_matches(m.get("id", ""), model):
                # Prefer loaded instance context (actual runtime value)
                for inst in m.get("loaded_instances", []):
                    cfg = inst.get("config", {})
                    ctx = cfg.get("context_length")
                    if ctx and isinstance(ctx, (int, float)):
                        return int(ctx)
                break    # <-- exits without checking m["max_context_length"]

Once the matching model entry is found and loaded_instances is empty, the inner for doesn't run, and break exits the outer loop. The function falls through to the OpenAI-compat /v1/models/{model} and /v1/models paths — neither of which carries context info on LM Studio — and ultimately returns None.

Proposed Fix

Inside the if server_type == "lm-studio": block, after the loaded_instances loop, fall back to m["max_context_length"] if it's present and positive, and only break (without returning) when neither is available. Roughly:

if server_type == "lm-studio":
    resp = client.get(f"{server_url}/api/v1/models")
    if resp.status_code == 200:
        data = resp.json()
        for m in data.get("models", []):
            if _model_id_matches(m.get("key", ""), model) or _model_id_matches(m.get("id", ""), model):
                # 1) Prefer the runtime ctx of a currently-loaded instance.
                for inst in m.get("loaded_instances", []):
                    cfg = inst.get("config", {})
                    ctx = cfg.get("context_length")
                    if ctx and isinstance(ctx, (int, float)):
                        return int(ctx)
                # 2) Otherwise fall back to the model's training-time max,
                #    which LM Studio always reports even when not loaded.
                max_ctx = m.get("max_context_length")
                if max_ctx and isinstance(max_ctx, (int, float)) and max_ctx > 0:
                    return int(max_ctx)
                break

Notes for reviewers:

• This complements [Bug]: fail to detect context length from remote LMStudio instances #47200's fix (which handles the remote-instance gating) — the two together let unloaded and remote LM Studio instances resolve correctly without manual config.
• The training-time max is an over-estimate of what JIT will actually load, so users who load with a smaller context still need a per-model context_length override under custom_providers[].models.<id> to be safe (see [Bug]: LM Studio custom_providers per-model context_length broken in 0.14.0 — regressed to 64K #30178). But returning the training max is strictly better than returning None — it bounds the budget instead of unbounding it via the family-default heuristic.
• The cache-bypass for provider == "lmstudio" at line ~1670 (# LM Studio is excluded — its loaded context length is transient...) is correct and should remain. The fix only affects the fallback ordering inside the probe itself.

Are you willing to submit a PR for this?

• I'd like to fix this myself and submit a PR

(I have a workaround in place — explicit per-model context_length: 8192 under custom_providers[].models.<id> — but that's a config-side band-aid, not a fix.)

[alt-glitch]

Related: #45037 (open PR) already adds a max_context_length fallback in _query_local_context_length() for LM Studio. That PR targets the "loaded at default 64K" case, but its post-loop max_context_length fallback also covers the empty-loaded_instances case described here (when no instance is loaded, loaded_ctx stays None and the max_context_length branch runs). So this is the same code path and fix — not a separate bug. Tracking as related so the maintainer can confirm #45037 fully resolves the unloaded-model path or extend it. (Distinct from #47200 remote-gating, #25989 manual-preload minimum, #30178 per-model override regression, as you noted.)

[GottZ]

Cross-referencing open PRs: 4 open PRs currently address or reference this issue.

• fix(agent): fall back to max_context_length in LM Studio probe #47698 (+6/-0) — fixes the root cause: the diff adds exactly the fallback to m.get("max_context_length") requested in this issue, inside the match block of _query_local_context_length after the loaded_instances loop, before the break; a minimal 6-line fix without tests, CI green.
• [codex] fix LM Studio context metadata fallback #59094 (+228/-49) — fixes the root cause: replaces the loaded_instances-only logic with _lmstudio_native_context_length(), which prefers loaded-instance context and otherwise falls back to max_context_length, applied in both the probe and the endpoint-metadata path, with test test_lmstudio_unloaded_model_uses_max_context_length; also covers the remote LM Studio case ([Bug]: fail to detect context length from remote LMStudio instances #47200).
• feat(lmstudio): add local model management #61606 (+2212/-131) — appears unrelated to this issue: it only changes model matching in _query_local_context_length_uncached (exact-key before alias) and still has no max_context_length fallback when the model is not loaded, so the cause described here remains; the PR body itself only declares "Related to [Bug]: LM Studio probe returns None when model not currently loaded — ignores max_context_length fallback #47678" and fixes [Bug]: Desktop UI model picker does not show custom/local provider models (hounddog_local / LM Studio) #48885 (model management / unload policy).
• fix(agent): guard against LM Studio silently serving a different model #64742 (+363/-2) — partial: it does not touch the probe but makes agent_init fail hard when the model is not loaded (check_lmstudio_model_served) and verifies response identity at runtime — this prevents running with an overestimated context, but does not implement the requested max_context_length fallback; in this issue's JIT scenario (model intentionally unloaded) the guard would block rather than fall back.

Duplicates: #47698 and #59094 implement essentially the same change.

Suggested consolidation: merge #59094 — it implements the same root-cause fix as #47698 (fallback to max_context_length when loaded_instances is empty), but as a reusable helper with type coercion, applied in both the probe and endpoint-metadata paths, and backed by tests including the unloaded case (both CI SUCCESS); it additionally closes #47200. #47698 could then be closed as a duplicate (a strict subset of #59094: same location, same approach, no tests). #61606 and #64742 should be evaluated independently of this issue — #61606 fixes #48885 and leaves the fallback unresolved, while #64742 is a complementary wrong-model guard that should be rebased/re-checked against the new fallback after #59094 lands. The issue itself can be closed with #59094.

Cross-PR triage: each diff was read against this issue; verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[GottZ]

This was generated by AI during triage.

Delta since our previous triage comment

Since our previous comment, @doncazper rebased #59094 onto current main and updated the remote native probe to require an LM Studio signature from /v1/models. The revised diff also adds focused unloaded-model and non-LM-Studio regression coverage, directly addressing the blocking concern in the visible keep_open review.

Changed pull requests

• [codex] fix LM Studio context metadata fallback #59094 best fix — (+210/-49) — best existing fix; keep open for maintainer re-review: despite the earlier keep_open review on [codex] fix LM Studio context metadata fallback #59094, the current diff now gates the native fallback on LM Studio identification, proves that non-LM-Studio endpoints are not probed, and retains loaded-runtime precedence with a tested max_context_length fallback for unloaded models.

Suggested consolidation

Recommendation changed: keep #59094 open as the best existing fix and request maintainer re-review rather than merge it now, consistent with the finding that this complex currently has no merge-lane-qualified PR.

Complex graph

flowchart TD
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I47200(["issue #47200 (open)"])
    I47678(["issue #47678 (open)"])
    P45037["PR #45037 (closed)"]
    subgraph Dup47244 ["PRs duplicating each other"]
        P47244["PR #47244 (open)"]
        P47693["PR #47693 (closed)"]
        P47698["PR #47698 (open)"]
        P51662["PR #51662 (closed)"]
        P59094["PR #59094 (open)"]
    end
    P45037 -.->|partial| I47678
    P47693 -->|fixes| I47678
    P47698 -->|fixes| I47678
    P51662 -->|fixes| I47678
    P59094 -->|best fix| I47678
    P47244 -.->|partial| I47200
    P59094 -->|best fix| I47200
    class I47200 open
    class I47678 open
    class I47678 target
    class P45037 closed
    class P47244 open
    class P47693 closed
    class P47698 open
    class P51662 closed
    class P59094 open
    class P59094 best
    class P59094 best
    click I47200 "https://github.com/NousResearch/hermes-agent/issues/47200"
    click I47678 "https://github.com/NousResearch/hermes-agent/issues/47678"
    click P45037 "https://github.com/NousResearch/hermes-agent/pull/45037"
    click P47244 "https://github.com/NousResearch/hermes-agent/pull/47244"
    click P47693 "https://github.com/NousResearch/hermes-agent/pull/47693"
    click P47698 "https://github.com/NousResearch/hermes-agent/pull/47698"
    click P51662 "https://github.com/NousResearch/hermes-agent/pull/51662"
    click P59094 "https://github.com/NousResearch/hermes-agent/pull/59094"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 6 pull requests and 2 issues in this complex. Each diff was read against this issue; Assessment working set: 37 kB of PR diffs, 18 kB of issue/PR text, 5 kB of discussion (11 comments), 9 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.