Desktop runtime readiness probes can saturate the shared RPC pool

[reymondmeking-dot]

Describe the bug

The Desktop periodically calls setup.runtime_check to decide whether inference is ready. That RPC is routed through the shared _LONG_HANDLERS executor. If provider resolution becomes slow because of GIL pressure, a blocked keyring, OAuth refresh, or filesystem/auth lookup, every overlapping Desktop poll starts another provider-resolution call and occupies another worker in the shared RPC pool.

During a long tool-heavy session this can exhaust the shared pool and delay unrelated JSON-RPC responses. The Desktop then reports that inference is not ready even though credentials are configured and the provider itself is healthy.

Observed log pattern:

event loop stalled 20-30s (GIL pressure suspected)
ws write slow (loop stalled >10.0s)
setup.runtime_check ... timed out / failed
setup.status reports configured credentials, but runtime resolution still failed

Switching model providers does not resolve the symptom because the contention is inside the gateway process rather than at a specific provider.

Environment

• Windows 11
• Python 3.11
• Hermes Desktop / Hermes Agent 0.18.0
• Reproduced with multiple custom OpenAI-compatible providers

Root cause

setup.runtime_check is correctly routed away from the WebSocket reader thread by PR #57335, but the shared RPC executor does not deduplicate in-flight readiness checks. A slow check can therefore overlap with later 15-second Desktop polls. Each duplicate probe consumes a shared worker while resolving the same provider state.

A deterministic test can reproduce the problem by making resolve_runtime_provider() block and issuing overlapping setup.runtime_check requests. Current main invokes the resolver once per request instead of sharing the existing in-flight probe.

Expected behavior

• At most one runtime-resolution probe should be active per requested provider.
• Later polls should reuse the in-flight probe instead of consuming more shared RPC workers.
• A transient timeout should be treated as an unknown readiness result, not ok=false, because setup.status may still authoritatively confirm that credentials are configured.
• Unrelated gateway RPCs should remain responsive.

Suggested fix

1. Run provider resolution on a dedicated single-worker executor instead of recursively consuming the shared RPC pool.
2. Keep one in-flight future per requested provider and reuse it for overlapping polls.
3. Allow the first caller a bounded wait; while the probe is still running, return a retryable result without an ok=false value.
4. Clear the single-flight entry when the background probe completes.

This preserves the intent of #57335—keeping the WebSocket reader responsive—while preventing frontend polling from starving the executor that serves the rest of the gateway.

[reymondmeking-dot]

Here is a deterministic reproduction against current main, plus the evidence from the affected Windows Desktop session.

Deterministic reproduction

This does not require real provider credentials or network access. It replaces resolve_runtime_provider() with a deliberately blocked function, then starts eight overlapping readiness requests—the same number as the default shared RPC pool size.

import os
import tempfile
import threading
import time

os.environ["HERMES_HOME"] = tempfile.mkdtemp(prefix="hermes-runtime-repro-")

from tui_gateway import server
import hermes_cli.main as main
import hermes_cli.runtime_provider as runtime_provider

release = threading.Event()
lock = threading.Lock()
entered = 0


def slow_resolve(requested=None):
    global entered
    with lock:
        entered += 1
    release.wait(timeout=5)
    return {
        "provider": "custom",
        "api_key": "no-key-required",
        "source": "reproduction",
    }


runtime_provider.resolve_runtime_provider = slow_resolve
main._has_any_provider_configured = lambda: True


def check(index):
    server.handle_request(
        {"id": str(index), "method": "setup.runtime_check", "params": {}}
    )


threads = [threading.Thread(target=check, args=(i,)) for i in range(8)]
for thread in threads:
    thread.start()

time.sleep(0.5)
print(f"resolver entries for 8 overlapping polls: {entered}")

release.set()
for thread in threads:
    thread.join(timeout=2)

Result on the old/current-main behavior:

resolver entries for 8 overlapping polls: 8

Each request performs the same resolution independently. In production these handlers are dispatched through _LONG_HANDLERS, so the eight blocked calls occupy all eight default tui-rpc workers.

Result with PR #65153:

resolver entries for 8 overlapping polls: 1

The first request owns the dedicated readiness probe. Later requests reuse the in-flight state and return a retryable result without ok=false.

Real session evidence

The issue was reproduced on Windows 11 / Python 3.11 / Hermes Agent 0.18.0 during a long Desktop task. The provider was not the bottleneck:

resolve_runtime_provider(): 0.027s
direct HTTPS response:       0.27s

At the same time, the gateway repeatedly logged:

event loop stalled 30.2s (GIL pressure suspected)
ws write slow (loop stalled >10.0s)
setup.status reports configured credentials, but runtime resolution still failed

The same symptom occurred after switching providers, which is consistent with gateway/RPC-pool contention rather than a provider-specific outage.

Fix verification

• 21 overlapping readiness requests shared exactly one provider-resolution probe.
• No overlapping request returned a false ok=false value.
• scripts/run_tests.sh tests/test_tui_gateway_server.py -k setup_runtime_check -q: 5 passed.
• scripts/run_tests.sh tests/tui_gateway/test_inline_rpc_gil_starvation.py -q: 8 passed.
• After installing the patch and restarting Desktop, more than three polling intervals completed without another runtime-check timeout or readiness failure.

I hope this is sufficient to reevaluate the needs-repro label. I can provide additional sanitized timing/log data if useful.

[DawnCloud1213]

Confirmed on a real Windows 10 host with additional findings. This is the same bug family as #60654 / #55433 (WS stall + request timed out on Desktop).

Live reproduction (real machine, Hermes desktop build 2026-07-23 / v0.19.0)

Connecting directly to the desktop backend (hermes serve, port 2903) via WebSocket and issuing the same RPCs the Desktop polls:

• setup.status → no response for 120s+ (intermittent)
• model.options → no response for 120s+ (intermittent)
• In between these episodes, the same RPCs return in ~0.0s — so this is a periodic starvation, not a slow handler.

Cycle pattern observed by the user: restarting the GUI fixes it; ~10–20 minutes later "Inference not ready" + request timed out: model.options return. The in-app "restart gateway" button does NOT fix it (the gateway process is separate from the desktop serve process that owns the RPC pool).

Confirms the #65151 mechanism

• tui_gateway/server.py:275 — 8-worker tui-rpc ThreadPoolExecutor, no queue bound; RPCs are _pool.submit(...) (server.py:1593), so when workers are saturated every subsequent RPC queues indefinitely.
• py-spy dumps during healthy windows show all 8 tui-rpc threads idle; the starvation is periodic and transient.
• Note: resolve_runtime_provider() itself does no network I/O (returns in ~0.01s even with a dead base_url), so provider-side outages are NOT the trigger — contention is inside the process, exactly as Desktop runtime readiness probes can saturate the shared RPC pool #65151 describes.

Additional aggravating factor found on this machine: desktop serve runs an embedded cron ticker

hermes_cli/web_server.py:147 _start_desktop_cron_ticker — the desktop backend embeds a cron scheduler (desktop is hermes dashboard, not gateway, so this exists to fire Desktop-created cron jobs). This user ALSO runs two gateways (default + eft-professor profiles, two QQ bots), so three processes compete for cron .tick.lock and kanban dispatcher locks. gateway.log shows repeated kanban dispatcher: another gateway already holds the dispatcher lock. The cron ticker + agent jobs (which invoke the LLM) inside the serve process are plausible contributors to the GIL pressure that makes readiness probes slow enough to pile up.

Suggested fix (supporting #65151)

Single-flight / dedicated single-worker executor for setup.runtime_check/setup.status probes (reuse one in-flight probe per provider), and consider bounding the shared RPC pool queue. Also worth checking: whether the desktop-embedded cron ticker should share the scheduler with a running gateway instead of running its own (lock contention + extra GIL load).

Environment: Windows 10, Hermes v0.19.0 (2026.7.20), desktop build 2026-07-23, Python 3.11.

[DawnCloud1213]

Precise root cause captured live with py-spy during an active episode (Windows 10, desktop backend PID 19480).

py-spy dump: all 8 tui-rpc workers stuck in the SAME network call

Thread "tui-rpc_0" .. "tui-rpc_7" (all 8):
  create_connection (urllib3/util/connection.py:73)
  _new_conn / connect / _make_request ...
  requests.get (requests/api.py:73)
  fetch_models_dev (agent/models_dev.py:293)     <- requests.get(MODELS_DEV_URL, timeout=15)
  _get_provider_models (agent/models_dev.py:422)
  get_model_capabilities (agent/models_dev.py:464)
  _apply_capabilities (hermes_cli/inventory.py:290)
  build_models_payload (hermes_cli/inventory.py:257)
  model.options handler (tui_gateway/server.py:14979)

MODELS_DEV_URL = https://models.dev/api.json (a ~3.3 MB catalog, fetched per-provider inside get_model_capabilities).

Why this saturates the pool (more precise than the original report)

The model.options RPC always passes capabilities=True -> for EVERY provider row it calls get_model_capabilities -> fetch_models_dev. When models.dev is slow/blocked from this network (TCP connect hangs — verified intermittently; measured 1.5s OK now, but connect can stall during episodes), each requests.get(timeout=15) blocks a worker for up to 15s. With several concurrent model.options calls (desktop retries / multiple picker opens), all 8 workers are occupied for the full 15s window, and every other RPC (setup.status, etc.) queues -> Desktop shows "Inference not ready" + request timed out: model.options. Then the requests time out, the pool drains, and the cycle repeats on the next model.options trigger — matching the user's ~10-20 min recurring pattern.

So the saturating call is the models.dev catalog fetch inside model.options capability enrichment, not resolve_runtime_provider itself.

Suggested fix direction (in addition to #65153)

• Single-flight + long TTL for the models.dev fetch (it is cached on disk for 1h already — the issue is the cold-fetch/refresh path holding pool workers), and/or
• Do capabilities enrichment outside the RPC thread pool (prefetch/cache) or with a much shorter timeout; and/or
• Bound concurrent models.dev fetches to 1 (all providers share the same catalog file — fetching it once per provider row is wasteful).

Environment: Windows 10, Hermes v0.19.0, desktop build 2026-07-23, Python 3.11.

[GottZ]

This was generated by AI during triage.

Summary

One of the three listed PRs addresses #65151: #65153 isolates and deduplicates runtime-readiness probes, while #55440 and #67524 modify only Anthropic image handling for #55432. The newer evidence also identifies an independent shared-pool starvation path through repeated models.dev fetches in model.options, so #65153 fixes the reported setup.runtime_check mechanism but not the entire observed symptom family.

Related pull requests

• fix(anthropic): normalize image media_type for Messages API (#55432) #55440 unrelated — (+33/-0) — unrelated: Changes only Anthropic data-URL media-type normalization for Anthropic image input: media_type forwarded verbatim, so image/jpg (and svg) data URLs are rejected with a 400 #55432 and does not touch the gateway RPC pool, setup.runtime_check, or model.options. Despite the keep_open review on fix(anthropic): normalize image media_type for Messages API (#55432) #55440, its useful JPEG-alias handling is incorporated into [codex] fix(anthropic): validate image media types #67524, while its unsupported-format fallback incorrectly relabels unchanged bytes as image/jpeg.
• [codex] fix(gateway): singleflight runtime readiness probes #65153 best fix — (+237/-26) — best existing partial fix: Adds a bounded runtime-check executor, same-provider single-flight reuse, unknown-on-timeout responses, and dispatch-level regressions that keep unrelated RPC work responsive. The current diff addresses the keep_open review on [codex] fix(gateway): singleflight runtime readiness probes #65153 by allowing distinct providers to run concurrently and testing through server.dispatch, but it does not change the independently observed model.options path that can saturate all eight shared workers through repeated models.dev fetches.
• [codex] fix(anthropic): validate image media types #67524 unrelated — (+164/-4) — unrelated: Fixes the separate Anthropic image input: media_type forwarded verbatim, so image/jpg (and svg) data URLs are rejected with a 400 #55432 image-media problem through alias normalization, byte sniffing, raster transcoding, and local rejection; it does not modify setup.runtime_check or model.options. Its current diff addresses the keep_open review on [codex] fix(anthropic): validate image media types #67524 by routing non-native or unrecognized bytes through PNG conversion or rejection regardless of the declared MIME type and adding mislabeled BMP, SVG, and corrupt-payload regressions.

Duplicates

#55440 and #67524 overlap on the separate #55432 image/jpg fix; #67524 is the corrected broader implementation, so #55440 is superseded by #67524. Neither is a duplicate of #65153.

Suggested consolidation

Keep #65153 open with a salvage path as the best existing fix for #65151: preserve its bounded executor, per-provider single-flight logic, unknown timeout semantics, and dispatch-level concurrency tests, while limiting its claim to setup.runtime_check and addressing the separately evidenced model.options/models.dev starvation path in follow-up work. Close #55440 as duplicate of #67524 (#55440 → #67524) because the latter retains the alias fix without the false JPEG relabeling; keep #67524 under #55432 rather than treating either image PR as part of the #65151 fix.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I55432(["issue #55432 (open)"])
    I65151(["issue #65151 (open)"])
    P55440["PR #55440 (open)"]
    P65153["PR #65153 (open)"]
    P67524["PR #67524 (open)"]
    P65153 -->|best fix| I65151
    P55440 -.->|partial| I55432
    P67524 -->|best fix| I55432
    class I55432 open
    class I65151 open
    class I65151 target
    class P55440 open
    class P65153 open
    class P67524 open
    class P65153 best
    class P67524 best
    click I55432 "https://github.com/NousResearch/hermes-agent/issues/55432"
    click I65151 "https://github.com/NousResearch/hermes-agent/issues/65151"
    click P55440 "https://github.com/NousResearch/hermes-agent/pull/55440"
    click P65153 "https://github.com/NousResearch/hermes-agent/pull/65153"
    click P67524 "https://github.com/NousResearch/hermes-agent/pull/67524"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 3 pull requests and 2 issues in this complex. Each diff was read against this issue; Assessment working set: 24 kB of PR diffs, 12 kB of issue/PR text, 14 kB of discussion (11 comments), 6 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.