SSH Remote app-server accumulates ~100k unreaped Git children during long-lived multi-client sessions

[CubeLander]

Version and environment

• Codex Desktop: 26.721.41059
• Remote Codex CLI during incident: 0.145.0
• Desktop client: macOS
• Remote: openEuler 24.03 container, Linux 5.15, aarch64
• Container PID 1: sshd -D

The remote CLI has since updated to 0.146.0, but Remote remains disabled and the incident has not been intentionally reproduced.

What happened

After roughly two days of a long-lived Codex Desktop SSH Remote connection, with several threads resumed or controlled from Desktop and Android Remote, the host became almost unusable.

Live inspection of the remote app-server found:

open file descriptors:  ~378,000
child processes:        ~125,000
zombie processes:       ~96,000–99,000
zombie command:         git (nearly all observed zombies)
zombie parent:          codex app-server

Restarting the app-server/container immediately restored normal responsiveness and removed the Codex-owned zombies.

Strongest diagnostic evidence

The structured app-server log shows that Codex Desktop continued issuing process/spawn requests while the server was degraded. It repeatedly launched a repository-state pipeline every few seconds, including:

git config --null --get core.fsmonitor
git ... status --porcelain=v1 -z
git ... diff --cached ... --name-only -z
git ... diff ... --raw --numstat -z
git ... hash-object --stdin-paths
git ... ls-files --others --exclude-standard -z
git ... rev-parse --verify --quiet HEAD^{commit}

Immediately after a clean restart, one directly observed app-server child was:

git -c core.hooksPath=/dev/null -c core.fsmonitor= \
  add --sparse --pathspec-from-file=- --pathspec-file-nul

This does not prove that every zombie had the same arguments, but it establishes both ownership and the active Git repository-state path.

Other contemporaneous signals included:

failed to refresh available models: timeout waiting for child process to exit
sampling stream disconnects and retries
SQLite statements exceeding the slow threshold
ignored world-state patch without a full snapshot during thread resume
shell snapshot command timeout
repeated Remote Control token refresh and WebSocket reconnect attempts

The large zombie/FD count was already present before the final reconnect storm, so the network failures may be an amplifier or consequence rather than the initial trigger.

Likely failure boundary

Confirmed:

• Codex Desktop repeatedly requested Git subprocesses through app-server process/spawn.
• Exited Git children accumulated under that app-server instead of being reaped.
• Process and FD counts grew without a useful bound.

Most likely:

• A repository refresh/reconciliation loop was duplicated or left alive across multi-client thread resume/reconnect activity.
• The process cleanup path failed to wait/reap children after timeout, cancellation, disconnect, or superseding refreshes.

The available evidence is sufficient to investigate the process/spawn ownership and repository-refresh lifecycle. We have not attempted to identify the exact source line because the incident was destructive enough to require a container restart.

Expected behavior and suggested fix direction

1. Every process/spawn child must be waited for and reaped on success, timeout, cancellation, client disconnect, thread resume, and shutdown.
2. Repository refreshes should be single-flight/coalesced per repository and have a strict concurrency bound.
3. Reconnecting or resuming the same thread must not duplicate background Git polling/reconciliation tasks.
4. Add a Linux soak test that repeatedly resumes/disconnects clients and asserts that child count, zombie count, and FD count return to baseline.

Reproduction shape

The exact trigger is intermittent:

1. Connect Codex Desktop to a Linux host with SSH Remote.
2. Keep app-server alive for an extended period with several long-running threads.
3. Resume/control threads from Desktop and a mobile Remote client.
4. Introduce Remote Control/Responses WebSocket failures or reconnects.
5. Monitor app-server children, zombies, and FDs over time.

We have not intentionally rerun this workload because the first occurrence produced approximately 100k zombies.

Related issues

• Codex Desktop app-server leaks child processes and writes excessive logs after crash/restart #26869
• [Persistent multi-version bug for VS Code SSH Linux] enters overload loop and collapse after threads stuck and recovery #20029
• Desktop Connections launches remote app-server with scrubbed env and leaves a stale backend that local CLI later reuses #29138
• codex exec hangs indefinitely — codex_models_manager logs "timeout waiting for child process to exit", no error surfaced, no recovery #34397
• TUI shows "Timeout waiting for child process to exit" when WebSocket transport fails; 75s startup delay before HTTP fallback #22634

This may share a lifecycle defect with #26869, but this report adds a Linux SSH Remote case with approximately 100k Git zombies, a single app-server parent, multi-client resume activity, and retained process/spawn evidence.