v0.3.8 — pool self-healing via per-client leases

Fixed

• An exhausted upstream connection pool now self-heals without a process restart (#12). A transient failure of the local forward proxy (e.g. Clash/Mihomo) can leave each failed handshake pinned as an ACTIVE connection inside httpcore; once the accumulation reaches the pool's 100-connection cap, every subsequent request fails with PoolTimeout until codexcomp is restarted — observed in the wild as 374 consecutive PoolTimeout errors over 9.5 hours. Now:
  • on PoolTimeout, the shared httpx.AsyncClient is rotated to a fresh generation under a lock (concurrent failures rotate a given generation only once); Codex's next retry uses the clean pool immediately;
  • on ConnectError, the client is rotated only when no other request holds a lease on it, so a dead handshake can't accumulate into later pool starvation;
  • client lifetime is tracked with per-client leases: an UpstreamRounds leases its pinned client from construction through the complete fold (every continuation round included), and a retired generation is closed only when its own lease count reaches zero — recovery never aborts a live stream or a fold in progress;
  • the failed request itself is not replayed, preserving the existing no-double-generation stance.
• The shared client is now closed on application shutdown.

Added

• /healthz reports client_generation, active_upstream_requests, and active_websockets.
• Transport-failure logs carry sanitized pool snapshots, proxy routing summaries (scheme/host/port only — never credentials), request ids, and timestamps.

Thanks @Anionex for the failure forensics in #12 and the fix (#13).

---

中文说明

修复上游连接池耗尽后必须重启进程才能恢复的问题(#12)。本地前置代理(如 Clash/Mihomo)瞬断时,每次失败的握手都会在 httpcore 里留下一条僵死的 ACTIVE 连接,累积到 100 连接上限后所有请求持续 PoolTimeout(实测 374 次、持续 9.5 小时)。现在 PoolTimeout 会在锁内把共享 client 轮换到新 generation(并发失败只轮换一次),Codex 的下一次重试立即使用干净连接池;ConnectError 仅在没有其他请求持有该 client 租约时轮换。client 生命周期按 per-client 租约管理:UpstreamRounds 从构造起租用其 pinned client 直到整个 fold(含全部续写轮)结束,退役的 generation 只在自身租约归零后关闭——恢复过程绝不中断进行中的流或 fold。失败请求本身不重放,维持不重复生成的既有立场。/healthz 新增 generation 与活跃计数字段,传输失败日志附带脱敏的池快照与代理路由摘要(仅 scheme/host/port,绝不含凭证)。感谢 @Anionex 的故障取证(#12)与修复(#13)。

v0.3.7 — clean terminals on upstream connection failures

Fixed

• Upstream transport failures no longer strand Codex in reconnect / "high demand" loops (#8). When the upstream — or a local forward proxy in front of it (e.g. Clash/Mihomo) — is temporarily unreachable (ConnectError, PoolTimeout, …), the proxy previously let the exception escape as a bare HTTP 500 or an abnormal WebSocket 1006 close, which Codex reads as backend trouble and keeps retrying. Now:
  • on the Responses path (WebSocket and POST SSE), the failure becomes a clean response.failed terminal with error.code: "upstream_connection_error";
  • on the /v1/* passthrough (e.g. GET /v1/models), it becomes a JSON 502 with the same code.
• Round-open failures now carry a string error.code (e.g. "upstream_429" instead of numeric 429), matching the string codes already used on the WebSocket path.

Deliberately no retry on transport failures: send() can fail after the request has already reached the upstream (ReadTimeout, RemoteProtocolError), so a blind re-send would risk a duplicate, double-billed generation.

Thanks @steveoon for the fix (#9).

---

中文说明

修复上游(或其前置的本地代理,如 Clash/Mihomo)临时不可用时,ConnectError/PoolTimeout 等传输错误以裸 HTTP 500 / WebSocket 1006 异常关闭逃逸,导致 Codex 误判为后端故障而陷入 "high demand" 重连循环的问题(#8)。现在 Responses 路径返回干净的 response.failed 终端事件(error.code: "upstream_connection_error"),/v1/* 透传路径返回 JSON 502;round-open 错误码统一为字符串形式(如 "upstream_429")。传输失败刻意不重试——请求可能已到达上游,盲目重发有重复生成、重复计费的风险。感谢 @steveoon 的贡献(#9)。

v0.3.6 — built-in candy A/B eval (codexcomp-eval)

The candy A/B eval now ships inside the package as the codexcomp-eval console script — installed users can measure the fix end-to-end right after pip install, no repo checkout needed.

What's new

• codexcomp-eval command: evals/candy_eval.py moved into the package as codexcomp/candy_eval.py (stdlib-only, logic unchanged) and is registered as a console script. Run codexcomp-eval -m gpt-5.5 -r xhigh -n 5 for a small grid or bare codexcomp-eval for the full 80-run grid; python -m codexcomp.candy_eval works too. Both on/off modes wire openai_base_url explicitly, so no config changes are needed for A/B runs.
• Docs: both READMEs document the built-in command, the results.jsonl resume behavior, the systemd-journal per-round fold prerequisite, and the coreutils timeout platform requirement (Linux/WSL; macOS needs coreutils).
• Credits: the candy puzzle task and its standalone-21 grading rule come from haowang02/codex-candy-eval — now acknowledged in both READMEs alongside the existing neteroster/CodexCont mechanism credit (attribution-only; the upstream repo declares no license).

No proxy data-path changes: fold/transport behavior is identical to 0.3.5.

---

中文说明

糖果 A/B 评测现已内置于包内：安装后直接运行 codexcomp-eval（小矩阵示例 codexcomp-eval -m gpt-5.5 -r xhigh -n 5），无需 clone 仓库；on/off 两种模式均显式传 openai_base_url，不依赖本机配置。两份 README 同步更新用法、断点续跑与平台前提说明，并新增对题目来源 haowang02/codex-candy-eval 的致谢（上游未声明 license，按来源归属致谢）。代理数据路径与 0.3.5 完全一致。

v0.3.5 — Zero-stall re-nudge on zero-reasoning continuation rounds

Fix

• Zero-reasoning continuation rounds are now treated as a zero-stall and re-nudged, instead of being let through on a 516-token total. Per josantonius's report on openai/codex#30364: under --max-n 1, a round 1 of 516 tokens triggers a continuation, but round 2 returning reasoning_tokens=0 was classified as clean and released — leaving the folded reasoning stuck at 516 and the answer still wrong.
• Round ≥2 with reasoning_tokens==0 is now a zero-stall (failed nudge / collapsed continuation): the fold appends another commentary nudge and keeps going, sharing the MAX_CONTINUE budget. Missing usage (None) and round-1 zero-reasoning are deliberately left untouched, matching the PR #5 stance that standalone zero-reasoning has no deterministic fingerprint.
• Budget exhaustion still flushes via the existing path tagged proxy_stopped_reason=max_continue — an answer is never dropped, and the four existing stop labels are byte-for-byte unchanged. test_fold.py adds three cases: stall-then-recover, stall-exhausted, and the round-1 zero-reasoning regression.

---

中文说明

本版修复续推轮的零推理 zero-stall：round ≥2 若返回 reasoning_tokens==0，视为 nudge 失败/续推塌缩，再追加一个 commentary nudge 继续，共享 MAX_CONTINUE 预算，不再让 516 token 的总量提前放行。缺失用量（None）与首轮零推理按 PR #5 立场刻意保持不变；预算耗尽仍走 max_continue 路径刷出，答案永不丢弃。

v0.3.4 — Transport-protocol fidelity: response-header passthrough + compaction-request bypass

Fix

• POST response-header passthrough: the upstream response's x-codex-* rate-limit snapshot family, x-models-etag, and request id are now mirrored onto the downstream SSE response (previously all dropped, so Codex couldn't see its rate-limit state behind the proxy). The POST handler first pulls the first event to force round 1 open, then answers downstream with round-1 response headers.
• Remote-compaction requests are never folded: a request whose input ends with a compaction_trigger item passes through as a single round, even if its usage happens to land right on the 518n−2 boundary — the trigger must be last and Codex requires exactly one compaction output item, both of which a continuation replay would break. Adds a new round verdict compaction_request. The community's real-world observation on openai/codex#30364 (zero-reasoning retries blowing out auto-compaction) independently confirms the necessity of this convergence.
• x-codex-turn-state sticky-routing token contract replay: if the upstream issues this header on a folded round 1, continuation requests carry it back verbatim (set-once mirroring Codex's per-turn semantics; not overridden when the client brings its own). In practice the current backend does not issue it, so this is dormant contract support.
• Rolls up the fixes accumulated since v0.3.3: deterministic teardown on the fold-abort path (drive_fold explicitly closes the generator, fold aborted downstream is always logged), the truncation-tier ceiling loosened (--max-n defaults to 0 = no ceiling, observed capturing up to n=13), global arguments placed before a CLI subcommand no longer being overridden by subparser defaults, and evals log-contract alignment.

Empirical basis (all real-hardware verified, codex 0.142.5 × gpt-5.5)

• Direct probe of the real backend: POST response headers carry the full rate-limit snapshot but not x-reasoning-included / x-codex-turn-state; the WS 101 handshake carries only x-models-etag. Accordingly the WS accept stays bare — declaring headers the upstream doesn't would skew Codex's context accounting away from a direct connection.
• The candy benchmark run in parallel across four effort tiers (low/medium/high/xhigh): round 1 was exactly 516 in all of them, and after folding all four tiers answered correctly (2–4 rounds, reasoning rescued at 4–12× the truncated amount).
• With a fold sandwiched in the middle of a tool loop, subsequent incremental frames on the same connection rebuilt correctly with continuous caching; codex exec resume had full context across connections; force-killing the client mid-generation had the proxy tear down deterministically and keep serving healthily; 4-way concurrency had no cross-talk.

Other

• Added CHANGELOG.md; documented the compaction_request round verdict; added test_ws.py to the development/contributing section.
• Each round adds a round open log line recording upstream-header presence (reasoning_included / turn_state / request id) for wire-level diagnosis.
• Versioning policy: a 0.0.1-level bump is the default release granularity (the unreleased 0.4.0 number was reeled back to 0.3.4).

---

中文说明

本版聚焦传输协议保真：把上游 POST 的 x-codex-* 限额快照家族、x-models-etag 与 request id 镜像回下游，让 Codex 在代理后仍能看到限额状态；对以 compaction_trigger 结尾的远程压缩请求单轮直通、永不折叠（新增 compaction_request 判定词）；并按契约回传 x-codex-turn-state sticky 路由令牌（当前后端未签发，属休眠支持）。同时并入 v0.3.3 后的若干修复，附带四档 effort 与工具循环/并发的真机实测依据。

v0.3.3 — Local WS stateful protocol implementation + transport-layer refactor

Fix

• Local implementation of Codex 0.142's stateful responses_websockets protocol. Each turn, Codex first sends a generate:false prewarm frame, and when the connection is reused it compresses the request into a previous_response_id + incremental (possibly empty) input. The old version forwarded these verbatim to the stateless SSE upstream, so prewarm frames were rejected with a 400 (Unsupported parameter: generate) and incremental frames were forwarded as if they were full input, making the model lose its history/context. Now WsSession (one per downstream connection) digests this protocol locally: prewarm frames are answered locally with a synthetic completed response (resp_codexcomp_prewarm_*); incremental frames are rebuilt to full input from last_input + last_output + delta before entering fold(); unknown ids fail loudly and close the connection, forcing Codex to reconnect with full input. Failed/aborted requests invalidate the stored state, matching Codex's behavior of only recording LastResponse on Completed. The upstream only ever sees stateless, full requests.
• Real-hardware verification (codex 0.142.5, official subscription): against v0.3.2 this reproduced 2 400s; the fixed version ran 63 requests with all 8 518n−2 folds completing, full context across multi-turn resume, and 0 failures.

Refactor

• Terminal events are now solely owned by fold(): when round 1 is rejected by the upstream, fold() mints its own response.failed, and RoundOpenError no longer escapes to the transport layer; the three response.incomplete emission sites are converged into a single helper. Every downstream terminal event shape now has exactly one owner, and the error paths are testable directly through the fold() interface.
• Extracted the drive_fold fold driver: the UpstreamRounds lifecycle (construction, iterating the fold, teardown/close) is collected into one shared async generator, and the SSE and WebSocket handlers degenerate into thin adapters that only do frame encoding, removing two copies of byte-for-byte duplicated driver logic. Along the way the accept-header "force-set then pop" round trip is dropped, and the passthrough path keeps the client's original accept.
• Explicit assembly: build_app(upstream_base) takes the parameter explicitly (falling back to the CODEXCOMP_UPSTREAM_BASE env var, then the official backend), removing the module-level global app and the "CLI writes the env var, import reads it" load-order coupling; DEFAULT_HOST / DEFAULT_PORT / DEFAULT_UPSTREAM are converged into a single source of truth in codexcomp/__init__.py.
• Removed --auto-port: a wired proxy must exclusively own the exact port that matches openai_base_url, so a port-drift branch contradicts the contract; when the port is busy it keeps failing loudly and exiting.

Tests

• Added test_ws.py: covers prewarm, empty incremental, tool-loop, unknown previous_response_id, and post-failure state invalidation.
• test_fold.py adds error-path cases: round 1 rejected → response.failed, continuation open failed → incomplete/upstream_error (reasoning retained, truncated message not released), and EOF with no terminal → incomplete/upstream_eof.

Logging additionally emits cached_tokens.

---

中文说明

本版在代理侧本地实现了 Codex 0.142 的有状态 responses_websockets 协议（prewarm 本地应答、增量帧重建为全量、未知 id 响亮失败），使上游始终只见无状态全量请求，真机 63 请求 8 次折叠全通过。同时把所有下游终局事件收归 fold() 唯一所有、提取共享的 drive_fold 驱动器、显式化 build_app 装配并移除 --auto-port，另补 test_ws.py 与 test_fold.py 错误路径用例。