Skip to content

Honcho memory prefetch hang on fresh CLI subprocess in v0.15.0 (regression from #27190) #34070

Closed as not planned

Description

@sadeegamhewa

Summary

In v0.15.0 (release/v0.15.0-strip-gui, commit b9e8b331c), fresh hermes chat -q subprocesses hang indefinitely in _ensure_workspace() of the Honcho memory plugin when called from a cold-start process (e.g. cron jobs spawning new subprocesses, Paperclip-style ticket dispatchers using hermes chat -q ... as a CLI invocation).

The hang is reproducible on v0.15.0 and absent on 973f27e95 (the prior pin), which is the commit that landed fix(run_agent): isolate background review fork from external memory plugins (#27190). That commit's title and #27190's framing suggest v0.15.0 may have regressed the isolation it introduced.

Environment

  • Hermes-agent: release/v0.15.0-strip-gui at b9e8b331c (released 2026-05-28)
  • Python: 3.11.15 (cpython, uv-managed)
  • OS: Linux (Hetzner CPX21 VPS, Ubuntu)
  • Honcho plugin: enabled, cloud-hosted backend, workspace already created days prior
  • Provider: OpenRouter (anthropic/claude-sonnet-4)
  • Deployment shape: hermes-gateway systemd user service + hermes chat -q ... subprocess invocations from external dispatchers (Paperclip) and cron jobs

Repro

  1. Pin Hermes at release/v0.15.0-strip-gui (b9e8b331c)
  2. Run uv sync, restart hermes-gateway
  3. Invoke a fresh CLI subprocess:
    timeout 30 hermes chat -q "respond with ok" -Q -m anthropic/claude-sonnet-4 --provider openrouter --yolo

Expected: "ok" returned within ~5 seconds.

Actual: Subprocess hangs in SSL.read until the timeout 30 kills it. Traceback at interrupt:

File ".../agent/conversation_loop.py", line 757, in run_conversation
  _ext_prefetch_cache = agent._memory_manager.prefetch_all(_query) or ""
File ".../agent/memory_manager.py", line 348, in prefetch_all
  result = provider.prefetch(query, session_id=session_id)
File ".../plugins/memory/honcho/__init__.py", line 582, in prefetch
  ctx = self._manager.get_prefetch_context(self._session_key)
File ".../plugins/memory/honcho/session.py", line 733, in get_prefetch_context
  user_ctx = self._fetch_peer_context(session.user_peer_id, ...)
File ".../plugins/memory/honcho/session.py", line 985, in _fetch_peer_context
  card = self._fetch_peer_card(peer_id, target=target)
File ".../plugins/memory/honcho/session.py", line 939, in _fetch_peer_card
  return self._normalize_card(getter(target=target) ...)
File ".../honcho/peer.py", line 468, in get_card
  self._honcho._ensure_workspace()
File ".../honcho/client.py", line 272, in _ensure_workspace
  self._http.post(routes.workspaces(), body={"id": self.workspace_id})
... → httpx → httpcore _receive_response_headers → SSL.read HANG

The hang is in the Honcho client's lazy workspace-ensure POST during the prefetch path's cold start, not in the LLM provider call.

Why the gateway-mediated path doesn't hang

Long-lived hermes-gateway runs (cron jobs that go through the gateway, Telegram conversations) work fine on v0.15.0. The hang is specific to fresh hermes chat -q subprocesses where the Honcho client has just been instantiated and _ensure_workspace() has never been called.

Hypothesis: gateway warms the Honcho connection pool on startup and reuses it; fresh subprocesses pay the cold-start _ensure_workspace POST and hang.

What works

  • 973f27e95 (fix(run_agent): isolate background review fork from external memory plugins (#27190)): fresh subprocess returns "ok" cleanly within ~3 seconds with all the same env / Honcho config.

Asks

  1. Confirmation of regression: is the isolation that #27190 introduced still intact in v0.15.0? If yes, what's the expected behavior for the fresh-subprocess Honcho prefetch path that hangs? If no, this is a regression to file under that commit's owners.
  2. Workaround config flag (if regression isn't easy to revert): could we get a memory.prefetch.fresh_subprocess: false (or equivalent) config knob so deployments running CLI-dispatcher patterns can skip the prefetch in cold-start contexts without disabling Honcho memory globally?
  3. Honcho client timeout: the hang is unbounded; consider adding a default timeout to the Honcho httpx client (e.g. 10s) so failures fail fast instead of hanging until upstream-imposed timeouts.

Related

  • Original 2026-05-27 update attempt that surfaced this: hangs were observed but call site wasn't yet identified. Today's repro on v0.15.0 isolates the specific path.
  • Codex TypeError fix (dc9d677d5) — the original reason for attempting the update; that fix is great and we want it. Just blocked from adopting by the Honcho prefetch hang.

Activity

  1. sadeegamhewa commented on May 28, 2026

    @sadeegamhewa
    Author

    Follow-up: same root cause manifests differently on 973f27e95 (v0.14) — Honcho client hangs while Honcho service is healthy

    Following up on this issue with new evidence gathered while attempting a Paperclip-dispatched workflow on the pinned 973f27e95 baseline (the commit titled fix(run_agent): isolate background review fork from external memory plugins (#27190)).

    TL;DR

    The hang we reported on v0.15.0 reproduces on 973f27e95 as 30-second timeouts on every Honcho client operation, not infinite hangs. Even with #27190's isolation, the cumulative timeouts (3 × 30s per subprocess attempt, then Paperclip's retry logic re-runs the cycle) consume the dispatch budget before any LLM call happens. Critically: Honcho's cloud API is healthy — it responds to /health in <50ms — so the bug is in Hermes' Honcho client, not the upstream service.

    Repro

    1. On 973f27e95 (latest pin tracked in our handover, with the Honcho memory plugin enabled and a workspace already provisioned days prior), dispatch a Paperclip ticket through a hermes_local adapter. The adapter spawns a fresh hermes chat -q ... subprocess per the standard hermes-paperclip-adapter pattern.

    2. The fresh subprocess does its init, then attempts Honcho's prefetch / dialectic queries. Each operation times out at 30s. Three operations per attempt = 90 seconds gone before the LLM client is ever touched.

    3. Paperclip's terminal-run recovery retries the assignment. Each retry spawns another subprocess; same Honcho timeout pattern repeats. After 3 attempts, Paperclip marks the ticket blocked:

      Paperclip automatically retried continuation for this assigned in_progress issue during terminal run recovery, but it still has no live execution path. Latest retry failure: timeout - Traceback... Moving it to blocked so it is visible for intervention.

    Log evidence

    Sample from a single subprocess attempt (3 consecutive runs all looked like this):

    22:54:46  Memory provider 'honcho' registered (5 tools)
    22:54:46  Initializing Honcho client (host: hermes, workspace: welchman)
    22:56:18  WARNING plugins.memory.honcho.session: Honcho session '...' add_peers failed (non-fatal): Request timed out after 30.0s
    22:57:50  WARNING plugins.memory.honcho.session: Honcho session '...' loaded (failed to fetch context: Request timed out after 30.0s)
    22:57:50  INFO  Memory provider 'honcho' activated
    22:57:50  INFO  conversation turn: session=... model=anthropic/claude-sonnet-4 provider=openrouter msg='Paperclip API safety rule: Use Authorization: Bearer ...'
    22:59:21  WARNING plugins.memory.honcho.session: Honcho dialectic query failed: Request timed out after 30.0s
    [subprocess killed by adapter timeoutSec=300 shortly after — no LLM call ever logged]
    

    Honcho service is healthy

    curl https://api.honcho.dev/health from the same VPS the subprocess runs on:

    HTTP 200  total 0.043s
    

    curl https://api.honcho.dev/ returns HTTP 401 total 0.061s (auth required at the root path; expected, but the endpoint is alive and fast).

    So the upstream Honcho service responds in tens of milliseconds. The Hermes Honcho client is the source of the 30s timeouts. That eliminates "Honcho is having an outage" as the cause.

    Hypotheses worth exploring upstream

    These are guesses from a user-side perspective; one of these may map to something obvious to the maintainers:

    1. HONCHO_API_KEY auth issue — if the client's auth token is expired/invalid, Honcho returns 401, and the client may be retrying with exponential backoff that hits our 30s read timeout. This would explain "API responds fast / client hangs slow" — successive 401s + backoff.
    2. TLS / HTTP protocol mismatch — httpx default vs Honcho's edge behavior. Fresh subprocess pays full handshake cost; pooled gateway connections don't.
    3. Connection pool init for fresh subprocesses — something about Honcho client initialization in a cold process gets stuck in a way that the gateway's long-lived client doesn't.

    Why this matters for the asks in the original issue

    The original issue asked for either (a) restoring the #27190 isolation that v0.15.0 regressed, or (b) a config flag to disable Honcho prefetch in fresh subprocesses. Ask (b) becomes more urgent given this finding. Even on a Hermes version that preserves #27190 isolation, the Honcho prefetch path is still being entered on fresh subprocesses — it just times out instead of hanging indefinitely. The end-user effect (Paperclip dispatches fail) is the same.

    A fresh-subprocess-prefetch-disable flag would let users in our deployment shape (Paperclip → hermes_local adapter → fresh hermes chat -q subprocess per ticket) opt out of the prefetch entirely and let the LLM call proceed without memory context for that one invocation. Honcho memory could still be populated via the long-lived gateway path.

    Happy to test any candidate fix or apply a debug patch and report back.

  2. added
    type/bugSomething isn't working
    P3Low — cosmetic, nice to have
    comp/pluginsPlugin system and bundled plugins
    tool/memoryMemory tool and memory providers
    on May 29, 2026
  3. sadeegamhewa commented on Jun 10, 2026

    @sadeegamhewa
    Author

    Confirmed resolved in v0.16.0 in our environment.

    Setup: Hermes is dispatched via many fresh hermes chat -q subprocesses (a Paperclip→Hermes adapter), with the Honcho memory provider enabled (recallMode: hybrid, writeFrequency: async).

    The hang is concurrency-dependent — a single fresh subprocess completes fine on both versions, so the repro is a concurrent burst, not a single invocation. Running 5 concurrent fresh hermes chat -q subprocesses against one shared HERMES_HOME:

    • v0.14.0: 2 of 5 jobs stalled ~73–77s (the agent turn itself was only ~8s — i.e. ~64s of silent blocking around it, empty stderr). This is the prefetch hang.
    • v0.16.0: 0 of 5 stalled (all completed in ≤20s).

    The "Honcho: make startup fail open" hardening (#24847 / #32213 / #35170) appears to resolve it. Suggest this can be closed.

    Separately, under the same concurrent-burst stressor we observed an unrelated intermittent SIGABRT on process exit (after the turn completes successfully) in both v0.14.0 and v0.16.0 — filing that as its own issue, as it's a distinct teardown crash rather than the prefetch hang tracked here.

  4. teknium1 commented on Jul 13, 2026

    @teknium1
    Collaborator

    Thanks for the detailed concurrency reproduction and for confirming the result in your Paperclip deployment. This automated hermes-sweeper review finds the reported foreground prefetch/startup hang implemented on current main.

    • f24b7ed9d95301c7f610241588dcf0589b900cd2 (fix: make Honcho startup fail open, merged as fix(honcho): make startup fail open #24847) moves context/hybrid session initialization to a daemon thread; plugins/memory/honcho/__init__.py:362 waits no more than 0.1 seconds.
    • plugins/memory/honcho/__init__.py:653 returns empty context while initialization is still in flight, and :677 uses the non-blocking context-prefetch path rather than the old synchronous fetch.
    • plugins/memory/honcho/session.py:666 runs the underlying context fetch in a daemon thread.
    • Regression coverage is present in tests/test_honcho_startup_fail_open.py:42 and :100.
    • Your June 10 follow-up independently confirmed that the five-process concurrent fresh-subprocess case no longer stalled in v0.16.0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Low — cosmetic, nice to havecomp/pluginsPlugin system and bundled pluginssweeper:implemented-on-mainSweeper: behavior already present on current maintool/memoryMemory tool and memory providerstype/bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions