Windows: sandbox broker wedges conversation thread across client restarts (deny-read ACLs -> silent hangs, spinning codex-windows-sandbox-setup); evidence of ACE accumulation (TokenDefaultDacl error 1344)

[Lut3ce]

Summary

On Windows, the interactive Codex client's sandbox execution broker can enter a wedged state in which every spawned operation (cmd, PowerShell, Node, git, filesystem writes, browser startup) hangs indefinitely and ignores timeouts. The wedge is conversation-thread-scoped and persists across a full client restart: restarting Codex and resuming the affected conversation re-attaches to the same broken broker, and subagents spawned inside that thread inherit it. Fresh codex exec processes and brand-new conversations on the same machine work normally during the entire incident.

Environment

• Codex CLI: codex-cli 0.144.6
• OS: Windows 11 Home, build 10.0.26200
• Sandbox modes affected: default sandbox in interactive sessions (workspace-write)

Original error

Before the hang state, the client surfaced:

windows sandbox: helper_unknown_error: apply deny-read ACLs

After that, all spawns hung silently.

Minimal reproduction (inside an affected thread)

• Command: cmd.exe /d /c echo sandbox-health-ok
• Directory: C:\tmp (any directory reproduces)
• Expected: immediate output
• Actual: no output; the process timeout is ignored; the outer tool has to terminate it

Isolation evidence

Probes run inside the affected conversation after a full client restart + thread resume:

Probe	Spawns a process?	Result
Root shell canary (cmd.exe /d /c echo ...)	yes	hung
Fresh subagent running the same canary	yes (inherits thread broker)	hung
Non-mutating apply_patch parser	no (in-process)	responded normally (~0.2 s)
External codex exec dispatches, same machine, same hour	yes (new process tree)	all completed (multiple long multi-hundred-k-token runs incl. git push, pytest, cargo)

Additional system-level evidence: each hung canary left behind an orphaned codex-windows-sandbox-setup process spinning CPU indefinitely (~6–7 CPU-minutes each at ~19 MB RSS before being killed manually). Three such processes accumulated, their start times matching the three hung probes to the second.

Follow-up evidence (same day, after recovery)

In a brand-new conversation on the same machine (after killing the orphaned helpers), the very first spawn failed fast with:

windows sandbox: runner failed during SpawnChild: SetTokenInformation(TokenDefaultDacl) failed: 1344

Win32 error 1344 is ERROR_ALLOTTED_SPACE_EXCEEDED ("No more memory is available for security information updates") — the token's fixed-size default-DACL buffer overflowed. An immediate retry of the same command succeeded. Together with the original apply deny-read ACLs failure, this suggests the sandbox setup accumulates ACEs (in the token default DACL and/or object ACLs) across runs until security-information writes start failing, and that the helper's handling of that failure is inconsistent: sometimes a fast error (recoverable), sometimes the indefinite busy-loop described above (unrecoverable for the thread). A reboot/fresh logon session appears to reset the accumulated state.

Interpretation

The conversation thread appears to hold a persistent binding to a sandbox/execution broker that crashed or deadlocked after the ACL error. Client restart does not recycle that binding when the thread is resumed; subagents inherit it. The per-spawn codex-windows-sandbox-setup helper then spins forever (busy-loop, not a blocked wait, given the CPU burn) instead of failing fast, which is why timeouts are never honored.

Impact

• An affected conversation can still converse (in-process operations work) but cannot execute anything, which is confusing to diagnose.
• Timeout settings are silently ignored, so automation on top of the client stalls indefinitely.
• Orphaned sandbox-setup helpers leak CPU until killed manually.

Workaround

1. Abandon the affected conversation; start a brand-new one (or use codex exec). Do not resume the wedged thread — resuming re-attaches the broken binding even after client restart/reinstall.
2. Kill leftover codex-windows-sandbox-setup processes (identifiable by abnormal cumulative CPU).
3. Health-check any new context with a cheap canary first: cmd.exe /d /c echo canary-ok.

Suggested fixes

• Recycle/re-create the sandbox broker binding on client restart rather than persisting it with the resumed thread.
• Make codex-windows-sandbox-setup fail fast (propagate helper_unknown_error instead of busy-looping) and honor the caller's timeout.
• Avoid unbounded ACE accumulation in the token default DACL / object ACLs across sandbox runs (root cause of error 1344).
• Surface a visible "execution broker unhealthy" state instead of silent hangs.

[github-actions]

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• Windows Desktop elevated sandbox hangs after ACL setup and leaks setup helpers; unelevated works (0.145.0-alpha.18) #33732

Powered by Codex Action

[bbingz]

Analysis (community)

Sharing a static code read against main that lines up with the reporter’s symptoms. I have not reproduced this on a Windows host; confidence is medium-high on the hang/timeout path (code-verified), and medium on the exact ACE-growth mechanics that produce helper CPU spin vs. error 1344 (consistent with design + issue evidence, not runtime-confirmed here).

Environment / versions (from report + related)

• This issue: codex-cli 0.144.6, Windows 11 10.0.26200, interactive workspace-write
• Related: #33732 — elevated sandbox hangs after ACL phase; orphaned codex-windows-sandbox-setup.exe; unelevated works (0.145.0-alpha.18 / Desktop)

Repro status

• Reporter evidence quality looks high: canary (cmd.exe /d /c echo …), isolation table (in-thread hang vs fresh codex exec OK), timestamp-matched high-CPU setup helpers, exact error strings, post-recovery TokenDefaultDacl / 1344.
• This comment: code-path analysis only (not a local Windows repro).

Observed vs expected

Observed	Expected
After helper_unknown_error: apply deny-read ACLs, every sandboxed spawn in the resumed thread hangs; tool timeouts appear ignored	Fail fast with an actionable setup/runner error; honor tool/timeout cancellation
Each hung probe leaves a spinning codex-windows-sandbox-setup orphan	Setup helper exits (success or structured failure); no unbounded CPU spin
Resume re-breaks the conversation; fresh codex exec / new threads work	Setup failures are process-local and recoverable without abandoning the thread
Later: SetTokenInformation(TokenDefaultDacl) failed: 1344 (ERROR_ALLOTTED_SPACE_EXCEEDED)	Default-DACL / object ACL growth stays within Windows allotment, or maps to a clear retryable setup error

Root-cause hypothesis

Primary hang mechanism (high confidence from code): every elevated sandboxed spawn goes through require_logon_sandbox_creds → always run_setup_refresh_… before the runner starts (identity.rs, comment: “Always refresh ACLs … via the setup binary”). The orchestrator then waits on codex-windows-sandbox-setup with no bound:

• run_setup_refresh_payload → Command::status() (unbounded child wait) — setup.rs
• elevated full setup → WaitForSingleObject(…, INFINITE) — setup.rs / run_setup_exe_payload
• concurrent same-payload waiters share run_setup_singleflight and block on a Condvar until the leader completes — also no timeout (SetupFlight::wait)

If the helper stalls (reporter: indefinite CPU after/around deny-read / ACL work; #33732: “setup binary completed” never logged after ACL phase), the blocking setup path never returns. Tool timeout_ms / cancellation appear to cover the post-setup capture path rather than racing the setup wait, so timeouts look “never honored” and each probe can leak another setup process.

Secondary durability factor (medium confidence): sandbox security state is intentionally persistent across runs:

• sync_persistent_deny_read_acls (deny_read_state.rs) — keeps deny-read ACEs for descendants that may outlive the launcher; state in .sandbox/deny_read_acl_state.json; per-principal revoke only
• restricted-token default DACL (token.rs set_default_dacl / create_token_with_caps_from) grants GENERIC_ALL to logon + Everyone + every capability SID — error string matches the report exactly: SetTokenInformation(TokenDefaultDacl) failed: {err}
• capability maps for cwd/write-roots (cap.rs) grow without an obvious global GC in the paths reviewed

That design can bloat object DACLs and enlarge default DACLs until Windows returns 1344, and maps untyped helper failures through HelperUnknownError — matching the first surfaced string helper_unknown_error: apply deny-read ACLs around sync_persistent_deny_read_acls(...).context("apply deny-read ACLs") in the setup Full path.

On “conversation-thread-scoped broker binding across restart”: I did not find a serialized broker IPC handle that would reattach after a full process death. More consistent explanations with the code:

1. process-local SETUP_FLIGHTS singleflight while a long-lived client/app-server process is still alive;
2. resumed conversation re-applying the same permission / deny-read / workspace-root profile → same hanging refresh payload;
3. leftover spinning helpers and/or bloated ACL state that re-trigger only for that spawn profile (fresh exec / new thread may differ enough to avoid the bad path).

Happy to be corrected if there is a durable binding I’m missing.

High-level fix outline (not a PR)

Small, layered changes (each independently shippable). Analysis only — not opening a PR unless the team invites one.

1. Fail-fast setup waits — replace unbounded Command::status / WaitForSingleObject(INFINITE) with a budget (and cancel on tool expiration); kill the setup child on timeout; map failures to structured SetupErrorCode (including a dedicated deny-read / ACL code instead of HelperUnknownError).
2. Wire cancellation through setup — check cancellation during refresh; on timeout, abort a stuck singleflight leader and remove the flight entry so waiters fail clearly instead of waiting forever.
3. Make refresh non-wedging under partial ACL failure — if apply deny-read fails, fail the spawn immediately with actionable text; do not leave subsequent identical payloads hanging; consider skipping full refresh when marker + ACL fingerprint is unchanged.
4. Bound ACE / SID growth — (a) cap or partition default-DACL entries so TokenDefaultDacl cannot exceed allotment (e.g. grant GENERIC_ALL only to a small fixed set if capabilities are not required on the default DACL — note the PowerShell pipeline/IPC comment in token.rs); (b) GC unused principals from deny_read_acl_state.json and revoke their ACEs carefully; (c) prune unused writable_root_by_path / workspace_by_cwd after inactivity; (d) surface 1344 as a retryable setup error with cleanup guidance.
5. Observability — surface “sandbox setup unhealthy” when refresh exceeds budget or orphans accumulate; do not silently hang the conversation.

Non-goals / risks

• Do not change seatbelt/bwrap/Linux paths, or broadly rewrite the capability-SID model.
• Short timeouts may false-fail on first elevated provisioning or large ACL walks — prefer a longer budget for full setup vs refresh.
• ACL GC must not revoke ACEs still needed by running descendants (the intentional reason deny-read state is sticky).
• Changing default-DACL composition needs regression coverage for PowerShell pipelines / IPC object creation.

Test ideas

Failing-first, if the team (or an invitee) implements:

1. Stub a setup helper that never exits → run_setup_refresh errors within timeout T and does not leave a permanent SETUP_FLIGHTS entry blocking a second call with the same key.
2. Singleflight recovery: leader hangs past timeout → waiters fail fast; subsequent call starts a new flight.
3. TokenDefaultDacl size: large N capability SIDs either succeeds under a reduced default-DACL policy or returns a structured error containing TokenDefaultDacl/1344 (not a hang).
4. deny-read GC: principal A then B on same path; after A is globally undesired, GC revokes A’s ACE and drops A from state without removing B.
5. E2E elevated (if Windows CI): inject deny-read apply failure → spawn returns a clear windows-sandbox helper error quickly; second canary does not hang; cancellation aborts the setup wait.
6. Windows Desktop elevated sandbox hangs after ACL setup and leaks setup helpers; unelevated works (0.145.0-alpha.18) #33732-class regression: setup that reaches write-ACE / ACL phase then stalls is killed and reported; no accumulation of helpers under repeated failed probes.

Questions for maintainers

1. Does a “full client restart” always tear down any long-lived app-server/daemon, or can SETUP_FLIGHTS / helper processes outlive the UI?
2. For this incident, was Windows sandbox elevated (default) vs unelevated? Issue describes interactive workspace-write; Windows Desktop elevated sandbox hangs after ACL setup and leaks setup helpers; unelevated works (0.145.0-alpha.18) #33732 isolates hang to elevated-only.
3. What deny-read paths/globs and write-root set size were active when apply deny-read first failed?
4. Did orphaned helpers hold ACL-related named objects, or hot-loop in SetEntriesInAclW / GetNamedSecurityInfo on bloated DACLs (stack/ETL / sandbox log would settle this)?
5. Is 1344 from too many ACEs in one default DACL, or from a corrupted/oversized ACL buffer on the base token?
6. Could the reporter share redacted .sandbox setup logs + deny_read_acl_state.json + rough capability-SID counts if still available?

---

Happy to refine this analysis with any corrections from the team. If maintainers later decide an external PR would help and invite one, I’d be glad to help implement a minimal slice (e.g. bounded setup wait + singleflight cleanup) — no expectation of that unless useful.

[bbingz]

Building on the root-cause analysis above with a finer blame pass on the wait origins and the current fix status. Traced against main as of 5dd992a (2026-07-24).

#32864 is an amplifier, not the origin of setup hangs. git log -S "SetupFlight" -- codex-rs/windows-sandbox-rs/src/setup.rs returns 3370181ec / #32864 (2026-07-13, "Coalesce concurrent Windows sandbox setup requests"), a single-file PR adding SetupFlight / Condvar / run_setup_singleflight with no timeout. Before #32864, each setup spawn ran its own unbounded wait, so one stuck helper hung only that tool call. #32864 made concurrent identical-payload spawns share one leader's unbounded wait - amplifying a per-spawn stall into a shared-payload wedge. I am not calling the whole hang class a "recent regression": the underlying unbounded waits predate #32864 by months.

Wait origins (finer blame pass). The setup helper waits (refresh Command::status() and elevated WaitForSingleObject(INFINITE)) originate in #7792 ("Elevated Sandbox 2", 13c0919bf, which created setup_orchestrator.rs, later refactored into setup.rs), not #24831. The parent of #24831 (cb9178e^) already has both .status() on the refresh/setup paths and WaitForSingleObject(sei.hProcess, INFINITE) on elevated setup. #24831 (2026-05-29, "Add Windows sandbox provisioning setup command") added provisioning / refresh_only plumbing, not the waits. I am not claiming #4905 as the elevated-setup-wait origin - git log -S "WaitForSingleObject" hits #4905 on a different path (sandbox process capture), not the setup-helper wait.

Wedge mechanic (process-local, payload-keyed). setup.rs:174-179 removes the SETUP_FLIGHTS entry only after the leader's run() returns; the leader's run() calls run_setup_refresh_payload -> Command::status() (setup.rs:362, unbounded) / elevated WaitForSingleObject(…, INFINITE). Concurrent same-payload waiters share run_setup_singleflight and block on a Condvar until the leader completes - also no timeout (SetupFlight::wait). So one stuck leader occupies the flight for that payload key until it returns. The singleflight is process-local + payload-keyed (it may manifest as a conversation wedge if the app-server process lives across turns, but the mechanism is not a durable conversation-broker binding).

Not fixed as of 5dd992a. No commits touch setup.rs after the 2026-07-22 issue date. main still has unbounded cmd.status(), WaitForSingleObject(INFINITE), and the timeout-less SetupFlight::wait. The two post-singleflight commits to setup.rs (dfd2d81 #34612, 999a715 #34613) only added stdin(Stdio::null()) and proxy routing - no timeout. #34629 "Harden Windows elevated sandbox startup" (2026-07-21) and #34624 "Terminate process trees" did not touch the wait/singleflight paths.

Existing test + its specific hole. setup.rs:1339 identical_setup_requests_share_one_in_flight_run (also added by #32864) covers only happy-path dedup (waiter joins leader, runs == 1). It does not exercise the stuck-leader / timeout-recovery path. A useful failing test would inject a same-key stuck leader (whose cmd.status() never returns) and assert a bounded wait recovers and the flight is cleaned up - distinct-payload keys already do not share a flight today, so that is not the right recovery criterion.