Team managers remain active for hours with unresolved Codex code-mode calls after PID resource exhaustion

[salim4n]

Summary

Two Codex-backed team managers remained in an active turn for more than 11 hours with unresolved code-mode exec calls wrapping team_read_messages. One manager accumulated 17 incoming messages while still showing the original turn as running.

At the start of the incident, the Linux PID controller rejected process/thread creation in the application's systemd cgroup. Codex also logged Resource temporarily unavailable (os error 11) and code-mode host startup/exit errors in that interval.

The resource rejection and unresolved calls are confirmed separately. The exact causal path is not yet reproduced or attributed to a particular manager's code-mode host. This may require a Codex fix rather than an AionCore transport fix; filing here to track the team-runtime impact and recovery behavior.

Environment

• AionUi Web CLI 2.1.61, Linux x86_64, systemd.
• AionCore 0.1.72 with the bounded streaming-preview backport from fix(stream): bound and coalesce live tool output previews #947. This was not an unmodified current-main deployment.
• Codex CLI 0.150.1, app-server transport, GPT-5.6 family.
• Two teams, approximately eleven configured members including their managers. Members run development tasks, including test/build subprocesses.
• Locally configured service TasksMax=512, counting both processes and threads. This is our deployment setting, not an asserted upstream default.
• 2 vCPU / 8 GiB host; service MemoryHigh=5 GiB, MemoryMax=6 GiB.

Observed sequence (not yet a deterministic minimal reproduction)

1. Run both teams concurrently and let managers assign development work.

2. Several member runtimes start near the same time.

3. The kernel reports cgroup: fork rejected by pids controller for the application's service.

4. Codex logs process creation failures with OS error 11. Two managers issue code-mode calls equivalent to:

   const r = await tools.mcp__aionui_team__team_read_messages({});
   text(r);

   One actual call supplied a message cursor. No private cursor or call/session identifiers are included here.

5. Both session traces have an outbound custom_tool_call named exec and no matching output for over 684 minutes. The calls began about 0.55 seconds apart, roughly two seconds after the kernel refusal.

6. Other Codex processes log code-mode host exited during handshake / code-mode host closed its stdout; a code-mode host also traps. Those PIDs have not been correlated conclusively with the two managers.

7. Manually cancelling one manager produces aborted by user after 41106.8s. Its next turn successfully reads team messages and completes. The other manager remains pending, with 17 incoming messages verified in the SQLite chat projection, until a later service crash/restart.

No automated retry or queue replay was used in the investigation.

Evidence boundary

• A read-only check of the real Codex session traces deterministically detects the missing results at a historical cutoff. This is incident verification, not a regression test or a synthetic reproduction of the cause.
• Earlier team_read_messages requests succeeded in milliseconds. No matching server-side tools/call invoked trace was found for the stalled calls. However, that log occurs after an awaited agent lookup; absence alone cannot locate the stall.
• AionCore's coordinator keeps an active batch until settlement; the Codex reader intentionally has no blanket mid-turn deadline. Neither behavior by itself proves a bug: long-running valid tools must remain supported.
• Separate later aionui-web SIGABRT exits are under investigation. Available records do not establish their cause or connect them to this stall. They should not be conflated with an OOM or with fix(stream): bound and coalesce live tool output previews #947.
• A service task-quota increase to 1,024 was applied without restarting the service or changing its memory limits. This is mitigation, not proof of recovery correctness.

Expected behavior / candidate acceptance criteria

• A confirmed failed execution host or failed nested dispatch must not leave an unfinishable tool/turn permanently active.
• Propagate a meaningful terminal error and settle the corresponding active work when failure is known.
• If the backend is merely quiet or an operation is legitimately long-running, do not kill it using an arbitrary overall turn timeout.
• Preserve pending messages and user data. Never silently replay a possibly side-effectful tool call whose completion is unknown.
• Add a regression test that distinguishes host/dispatch failure from a healthy long-running operation, and verifies the real team coordinator can recover after explicit interruption.

Related reports reviewed

• [Windows Desktop 26.803.10989.0] codex-code-mode-host 0xc0000409 crash leaves concurrent exec calls stuck until restart openai/codex#38203 reports outstanding code-mode calls left pending after a host crash, on Windows Desktop. Similar symptom; different platform and unproven common cause.
• MCP tool approval elicitation can pend forever with no timeout or turn-state signal (headless app-server clients hang silently); rmcp is_closed() misses self-terminated service tasks openai/codex#39149 describes an unanswered MCP approval elicitation in a headless client. Also a possible indefinite-wait mechanism, but not demonstrated in this incident.
• feat(team): team mode reliability improvements, model switch persistence and runtime restart #787 concerns team read-message behavior; the observed call here never returned a result in the Codex trace.

Private prompts, task contents, domains, credentials, workspace paths, team names and session IDs are intentionally omitted.

[salim4n]

Isolated checks after the initial report

I ran official Codex 0.150.1 binaries in local Linux containers with no network access, real code-mode execution, a synthetic Responses API, and a read-only MCP fixture. Both CLI execution and the stdio app-server path were exercised.

• Control: MCP result and terminal turn received.
• Kill the code-mode host during a pending MCP call: the app-server forwarded code-mode host closed its stdout in about 6 ms and emitted turn/completed.
• Actually exhaust a cgroup PID limit before dispatch: the kernel's pids.events incremented, the host spawn failed with OS error 11, and app-server forwarded that error in about 57 ms and emitted turn/completed.
• The exhausted app-server did not exit within the harness's 25-second shutdown deadline and was killed by the harness. That is a separate shutdown symptom, not a reproduction of the indefinitely active turn.

These tests did not reproduce the reported manager freeze. They use Linux arm64 (the incident host is x86_64), synthetic model responses with explicitly enabled code-mode, a single fresh session, and a fixture MCP server rather than the full AionCore coordinator and real production conversation. They do not justify claiming a causal fix or adding a blanket turn timeout.

Next evidence needed: a bounded, privacy-filtered capture of app-server requests/notifications, nested-dispatch lifecycle and corresponding process state during an actual recurrence. No corrective PR is being claimed as validated at this stage.