Repository navigation
Description
Summary
In v0.15.0 (release/v0.15.0-strip-gui, commit b9e8b331c), fresh hermes chat -q subprocesses hang indefinitely in _ensure_workspace() of the Honcho memory plugin when called from a cold-start process (e.g. cron jobs spawning new subprocesses, Paperclip-style ticket dispatchers using hermes chat -q ... as a CLI invocation).
The hang is reproducible on v0.15.0 and absent on 973f27e95 (the prior pin), which is the commit that landed fix(run_agent): isolate background review fork from external memory plugins (#27190). That commit's title and #27190's framing suggest v0.15.0 may have regressed the isolation it introduced.
Environment
- Hermes-agent:
release/v0.15.0-strip-guiatb9e8b331c(released 2026-05-28) - Python: 3.11.15 (cpython, uv-managed)
- OS: Linux (Hetzner CPX21 VPS, Ubuntu)
- Honcho plugin: enabled, cloud-hosted backend, workspace already created days prior
- Provider: OpenRouter (
anthropic/claude-sonnet-4) - Deployment shape: hermes-gateway systemd user service +
hermes chat -q ...subprocess invocations from external dispatchers (Paperclip) and cron jobs
Repro
- Pin Hermes at
release/v0.15.0-strip-gui(b9e8b331c) - Run
uv sync, restart hermes-gateway - Invoke a fresh CLI subprocess:
timeout 30 hermes chat -q "respond with ok" -Q -m anthropic/claude-sonnet-4 --provider openrouter --yolo
Expected: "ok" returned within ~5 seconds.
Actual: Subprocess hangs in SSL.read until the timeout 30 kills it. Traceback at interrupt:
File ".../agent/conversation_loop.py", line 757, in run_conversation
_ext_prefetch_cache = agent._memory_manager.prefetch_all(_query) or ""
File ".../agent/memory_manager.py", line 348, in prefetch_all
result = provider.prefetch(query, session_id=session_id)
File ".../plugins/memory/honcho/__init__.py", line 582, in prefetch
ctx = self._manager.get_prefetch_context(self._session_key)
File ".../plugins/memory/honcho/session.py", line 733, in get_prefetch_context
user_ctx = self._fetch_peer_context(session.user_peer_id, ...)
File ".../plugins/memory/honcho/session.py", line 985, in _fetch_peer_context
card = self._fetch_peer_card(peer_id, target=target)
File ".../plugins/memory/honcho/session.py", line 939, in _fetch_peer_card
return self._normalize_card(getter(target=target) ...)
File ".../honcho/peer.py", line 468, in get_card
self._honcho._ensure_workspace()
File ".../honcho/client.py", line 272, in _ensure_workspace
self._http.post(routes.workspaces(), body={"id": self.workspace_id})
... → httpx → httpcore _receive_response_headers → SSL.read HANG
The hang is in the Honcho client's lazy workspace-ensure POST during the prefetch path's cold start, not in the LLM provider call.
Why the gateway-mediated path doesn't hang
Long-lived hermes-gateway runs (cron jobs that go through the gateway, Telegram conversations) work fine on v0.15.0. The hang is specific to fresh hermes chat -q subprocesses where the Honcho client has just been instantiated and _ensure_workspace() has never been called.
Hypothesis: gateway warms the Honcho connection pool on startup and reuses it; fresh subprocesses pay the cold-start _ensure_workspace POST and hang.
What works
973f27e95(fix(run_agent): isolate background review fork from external memory plugins (#27190)): fresh subprocess returns "ok" cleanly within ~3 seconds with all the same env / Honcho config.
Asks
- Confirmation of regression: is the isolation that
#27190introduced still intact in v0.15.0? If yes, what's the expected behavior for the fresh-subprocess Honcho prefetch path that hangs? If no, this is a regression to file under that commit's owners. - Workaround config flag (if regression isn't easy to revert): could we get a
memory.prefetch.fresh_subprocess: false(or equivalent) config knob so deployments running CLI-dispatcher patterns can skip the prefetch in cold-start contexts without disabling Honcho memory globally? - Honcho client timeout: the hang is unbounded; consider adding a default timeout to the Honcho httpx client (e.g. 10s) so failures fail fast instead of hanging until upstream-imposed timeouts.
Related
- Original 2026-05-27 update attempt that surfaced this: hangs were observed but call site wasn't yet identified. Today's repro on v0.15.0 isolates the specific path.
- Codex TypeError fix (
dc9d677d5) — the original reason for attempting the update; that fix is great and we want it. Just blocked from adopting by the Honcho prefetch hang.
Activity
sadeegamhewa commentedon May 28, 2026
Follow-up: same root cause manifests differently on
973f27e95(v0.14) — Honcho client hangs while Honcho service is healthyFollowing up on this issue with new evidence gathered while attempting a Paperclip-dispatched workflow on the pinned
973f27e95baseline (the commit titledfix(run_agent): isolate background review fork from external memory plugins (#27190)).TL;DR
The hang we reported on v0.15.0 reproduces on
973f27e95as 30-second timeouts on every Honcho client operation, not infinite hangs. Even with#27190's isolation, the cumulative timeouts (3 × 30s per subprocess attempt, then Paperclip's retry logic re-runs the cycle) consume the dispatch budget before any LLM call happens. Critically: Honcho's cloud API is healthy — it responds to/healthin <50ms — so the bug is in Hermes' Honcho client, not the upstream service.Repro
On
973f27e95(latest pin tracked in our handover, with the Honcho memory plugin enabled and a workspace already provisioned days prior), dispatch a Paperclip ticket through ahermes_localadapter. The adapter spawns a freshhermes chat -q ...subprocess per the standard hermes-paperclip-adapter pattern.The fresh subprocess does its init, then attempts Honcho's prefetch / dialectic queries. Each operation times out at 30s. Three operations per attempt = 90 seconds gone before the LLM client is ever touched.
Paperclip's terminal-run recovery retries the assignment. Each retry spawns another subprocess; same Honcho timeout pattern repeats. After 3 attempts, Paperclip marks the ticket
blocked:Log evidence
Sample from a single subprocess attempt (3 consecutive runs all looked like this):
Honcho service is healthy
curl https://api.honcho.dev/healthfrom the same VPS the subprocess runs on:curl https://api.honcho.dev/returnsHTTP 401 total 0.061s(auth required at the root path; expected, but the endpoint is alive and fast).So the upstream Honcho service responds in tens of milliseconds. The Hermes Honcho client is the source of the 30s timeouts. That eliminates "Honcho is having an outage" as the cause.
Hypotheses worth exploring upstream
These are guesses from a user-side perspective; one of these may map to something obvious to the maintainers:
Why this matters for the asks in the original issue
The original issue asked for either (a) restoring the
#27190isolation that v0.15.0 regressed, or (b) a config flag to disable Honcho prefetch in fresh subprocesses. Ask (b) becomes more urgent given this finding. Even on a Hermes version that preserves#27190isolation, the Honcho prefetch path is still being entered on fresh subprocesses — it just times out instead of hanging indefinitely. The end-user effect (Paperclip dispatches fail) is the same.A fresh-subprocess-prefetch-disable flag would let users in our deployment shape (Paperclip → hermes_local adapter → fresh
hermes chat -qsubprocess per ticket) opt out of the prefetch entirely and let the LLM call proceed without memory context for that one invocation. Honcho memory could still be populated via the long-lived gateway path.Happy to test any candidate fix or apply a debug patch and report back.
sadeegamhewa commentedon Jun 10, 2026
Confirmed resolved in v0.16.0 in our environment.
Setup: Hermes is dispatched via many fresh
hermes chat -qsubprocesses (a Paperclip→Hermes adapter), with the Honcho memory provider enabled (recallMode: hybrid,writeFrequency: async).The hang is concurrency-dependent — a single fresh subprocess completes fine on both versions, so the repro is a concurrent burst, not a single invocation. Running 5 concurrent fresh
hermes chat -qsubprocesses against one sharedHERMES_HOME:The "Honcho: make startup fail open" hardening (#24847 / #32213 / #35170) appears to resolve it. Suggest this can be closed.
teknium1 commentedon Jul 13, 2026
Thanks for the detailed concurrency reproduction and for confirming the result in your Paperclip deployment. This automated hermes-sweeper review finds the reported foreground prefetch/startup hang implemented on current main.
f24b7ed9d95301c7f610241588dcf0589b900cd2(fix: make Honcho startup fail open, merged as fix(honcho): make startup fail open #24847) moves context/hybrid session initialization to a daemon thread;plugins/memory/honcho/__init__.py:362waits no more than 0.1 seconds.plugins/memory/honcho/__init__.py:653returns empty context while initialization is still in flight, and:677uses the non-blocking context-prefetch path rather than the old synchronous fetch.plugins/memory/honcho/session.py:666runs the underlying context fetch in a daemon thread.tests/test_honcho_startup_fail_open.py:42and:100.