[Codex App][Remote SSH] Thread hydration blocks the app-server queue and causes an endless reconnect loop

[yufeinever]

What version of the Codex App are you using (From "About Codex" dialog)?

26.721.11231.0; remote Codex CLI/app-server 0.146.0.

What subscription do you have?

ChatGPT subscription (exact tier not confirmed).

What platform is your computer?

Microsoft Windows NT 10.0.26200.0 x64; remote host is Linux 6.8.0-136-generic x86_64.

What issue are you seeing?

Codex Desktop Remote SSH can enter an endless reconnect loop when it automatically hydrates several existing threads. One slow thread/resume request occupies an app-server request slot for 50-65 seconds. Additional hydration and background requests fill all six in-flight slots, queued requests expire after 30 seconds, and the whole WebSocket eventually closes with code 1006.

This affects the entire remote host connection, not only the problematic thread. New threads cannot be created or sent while the queue is blocked. Normal SSH sessions remain stable, the managed remote app-server PID remains alive, and server CPU/memory are not saturated.

The affected session files were not exceptionally large by storage standards (approximately 2-20 MB), but some JSONL records were monolithic: individual records were approximately 1.2-1.64 MB. A 2.28 MB session with only 111 records contained one 1.26 MB record. This suggests that record shape and concurrent hydration, rather than only total session size, trigger the failure.

Representative redacted Desktop log sequence:

app_server_client_request_queue_rejected inFlightRequestCount=6
  method=plugin/list queueWaitMs=30003 reason=queue-expired

Request failed method=thread/resume durationMs=65397
  error="Codex app-server is not available"

app_server_connection.closed code=1006 transport=websocket
proxy_command_failed signal=SIGTERM operation=app_server_proxy

SIGTERM is logged after the WebSocket closes and appears to be cleanup, not the initial cause.

What steps can reproduce the bug?

1. Configure Codex Desktop on Windows with a Remote SSH Linux project.
2. Create several threads in that project, including threads containing large tool outputs or other large individual JSONL records.
3. Leave Remote SSH auto-connect enabled and keep those threads active/unarchived.
4. Restart Codex Desktop or reconnect the Remote SSH host so Desktop hydrates multiple threads.
5. Observe several concurrent thread/resume calls. One or more resume calls take tens of seconds.
6. Observe inFlightRequestCount=6, request queue expiration after 30 seconds, WebSocket close code 1006, and automatic reconnect.
7. Try to create or send a new thread while this is happening. The new request is blocked behind hydration requests or fails when the app-server connection closes.

The same SSH route becomes responsive immediately after the recovery steps below, without changing the network.

What is the expected behavior?

• A slow or malformed thread should only fail or delay that thread's hydration.
• Interactive operations such as thread/start and turn/start should not be blocked behind bulk hydration.
• Thread history should be streamed, paginated, or processed with bounded record sizes.
• Remote reconnect should be bounded and should not repeatedly replay the same failing hydration workload.
• The UI should identify the offending thread and offer a reversible recovery action.

Additional information

The effective recovery was:

1. Identify the thread whose thread/resume is blocking the queue.
2. Archive the offending large/abnormal thread or threads (reversible; do not delete them).
3. Cancel their pending resume operations and clear queued hydration requests.
4. Stop automatic reconnect instead of immediately replaying the same workload.
5. Restart the managed remote daemon once to clear already in-flight server requests, then reconnect only after the offending threads are archived.

Recommended product behavior: after a hydration timeout or repeated queue saturation, show an Archive task and stop reconnecting recovery action. It should archive the identified thread, cancel pending hydration, clear the affected queue, and pause automatic reconnect until the user explicitly retries. Interactive requests should also receive priority over background hydration.

After applying the recovery above, request latency on the unchanged connection returned to approximately 1-350 ms, turn/start completed in approximately 122 ms, and the connection remained stable. Before recovery, even smaller threads took 25-31 seconds because they were queued behind the blocking resume.

Related but not duplicate:

• Codex App Remote SSH is slow and fragile behind NLB compared with VS Code Remote SSH #20128 tracks Remote SSH fragility behind a network gateway/NLB.
• Codex Desktop Remote SSH reconnect loop leaks thousands of orphaned app-server processes when hosts share an NFS home #35217 tracks leaked app-server processes during reconnects with a shared NFS home.

This report is specifically about thread hydration/backpressure causing host-wide request starvation and a reconnect loop.

[github-actions]

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• [Windows][26.721.3404.0] App freezes on startup while restoring SSH remote threads #35096

Powered by Codex Action

[yufeinever]

I reviewed #35096. There is clear overlap: both are triggered by Windows Codex Desktop automatically restoring Remote SSH threads.

However, #36189 captures a narrower backend failure mode that #35096 does not document: thread/resume requests saturate all six app-server request slots, queued requests expire after 30 seconds, the WebSocket closes with code 1006, and automatic reconnect repeatedly replays the same workload. The UI does not necessarily become fully unresponsive, but new threads cannot be sent.

#36189 also includes request timings, evidence of monolithic 1.2-1.64 MB JSONL records, and a reproducible recovery: archive the offending threads, cancel hydration, stop reconnecting, clear the queue, and restart the managed daemon once.

These issues may share the same underlying thread-restoration regression, but I am leaving #36189 open as the app-server/backpressure and reconnect-loop case unless maintainers confirm they should be consolidated.

[tpn]

Confirming this on a Linux server & Windows ChatGPT App setup, Windows
connecting to the Linux box via "Connections" & ssh, with server-side evidence
from three related incidents. Remote host: Ubuntu (kernel 6.17), managed
standalone app-server 0.145.0 and later 0.146.0; client: Codex/ChatGPT Desktop
on a separate Windows workstation over SSH; current version 26.727.51351 but
this occurred recently on earlier versions over the past week.

Symptom match. Desktop entered the endless reconnect loop described here:
SSH itself fine, but Codex never finished connecting. At the time of the most
recent incident the remote session store was 16 GB across 6,621 rollout files —
largest 492 MB, with 61 files ≥ 10 MB. These are far past the 2–20 MB files
reported above, which may explain why our failure mode progressed further
(below).

Server-side observations across the three incidents (Jul 25, Jul 28, Aug 2):

• Each Desktop reconnect attempt spawns codex app-server proxy on the remote
  host via SSH. When the app-server stops responding, the proxies never time out
  and never exit. We accumulated 4,076 orphaned proxy processes in one
  incident and 3,506 in another (spawn cadence ~every 10–50 s), with the
  app-server's unix-socket accept queue saturated at 2,731 and 2,906 pending
  connections respectively. Same leak shape as Codex Desktop Remote SSH reconnect loop leaks thousands of orphaned app-server processes when hosts share an NFS home #35217, no NFS involved.
• The app-server itself failed two different ways: once spinning on futexes at
  ~7 cores (~46 k futex syscalls/s, SIGTERM ignored, required SIGKILL — matches
  app-server: degraded broker catches SIGTERM then deadlocks on a futex (needs SIGKILL); healthy broker exits <1s #34142), once fully deadlocked at 0% CPU while holding the listener. RSS
  reached 7.6 GB with another 3.8 GB in swap before the hangs.
• Between incidents the managed app-server also died silently and nothing
  restarted it (matches Persistent app-server and pid-update-loop disappear without restart, leaving stale daemon state on Linux #35295); the next incoming proxy then auto-spawned an
  unmanaged replacement (ppid 1) that the daemon refuses to manage, so recovery
  required manually killing it.

On the six-slot limit. I searched codex-rs (0.146.0) for the limiter: there
is no 6-permit semaphore anywhere in the app-server — request pools are 256/1024
and the per-connection ConnectionRpcGate is unbounded — and the string
inFlightRequestCount doesn't appear in this repo. So the six in-flight slots
and the 30 s queue expiry appear to live in the closed-source Desktop client,
meaning affected users have no configuration or self-build escape hatch on
either end.

Mitigation data point. Archiving old/large rollouts (active set reduced from
16 GB / 6,621 files to ~4.7 GB, largest active file now ~10 MB) stopped the
reconnect loop immediately. That's consistent with the hydration-cost theory:
six slots are fine when thread/resume is fast; they're fatal when a handful of
resumes take 50+ s.

Requests:

1. Make the Desktop client's in-flight request limit and queue expiry
   configurable — or better, deprioritize bulk thread hydration behind
   interactive requests so a slow resume can't starve new-thread creation.
2. Paginate or lazily hydrate thread/resume (0.146.0 already added paginated
   history for forks; resume seems like the natural next candidate).
3. Have codex app-server proxy fail fast (bounded connect/handshake timeout,
   then exit non-zero) when the app-server is unresponsive, so a wedged server
   degrades into fast failures instead of thousands of orphaned processes and a
   saturated accept queue.

Happy to provide the futex traces, accept-queue captures, or process trees from
any of the three incidents if useful.