Skip to content

v0.3.8 — pool self-healing via per-client leases

Latest

Choose a tag to compare

@dzshzx dzshzx released this 17 Jul 02:49

Fixed

  • An exhausted upstream connection pool now self-heals without a process restart (#12). A transient failure of the local forward proxy (e.g. Clash/Mihomo) can leave each failed handshake pinned as an ACTIVE connection inside httpcore; once the accumulation reaches the pool's 100-connection cap, every subsequent request fails with PoolTimeout until codexcomp is restarted — observed in the wild as 374 consecutive PoolTimeout errors over 9.5 hours. Now:
    • on PoolTimeout, the shared httpx.AsyncClient is rotated to a fresh generation under a lock (concurrent failures rotate a given generation only once); Codex's next retry uses the clean pool immediately;
    • on ConnectError, the client is rotated only when no other request holds a lease on it, so a dead handshake can't accumulate into later pool starvation;
    • client lifetime is tracked with per-client leases: an UpstreamRounds leases its pinned client from construction through the complete fold (every continuation round included), and a retired generation is closed only when its own lease count reaches zero — recovery never aborts a live stream or a fold in progress;
    • the failed request itself is not replayed, preserving the existing no-double-generation stance.
  • The shared client is now closed on application shutdown.

Added

  • /healthz reports client_generation, active_upstream_requests, and active_websockets.
  • Transport-failure logs carry sanitized pool snapshots, proxy routing summaries (scheme/host/port only — never credentials), request ids, and timestamps.

Thanks @Anionex for the failure forensics in #12 and the fix (#13).


中文说明

修复上游连接池耗尽后必须重启进程才能恢复的问题(#12)。本地前置代理(如 Clash/Mihomo)瞬断时,每次失败的握手都会在 httpcore 里留下一条僵死的 ACTIVE 连接,累积到 100 连接上限后所有请求持续 PoolTimeout(实测 374 次、持续 9.5 小时)。现在 PoolTimeout 会在锁内把共享 client 轮换到新 generation(并发失败只轮换一次),Codex 的下一次重试立即使用干净连接池;ConnectError 仅在没有其他请求持有该 client 租约时轮换。client 生命周期按 per-client 租约管理:UpstreamRounds 从构造起租用其 pinned client 直到整个 fold(含全部续写轮)结束,退役的 generation 只在自身租约归零后关闭——恢复过程绝不中断进行中的流或 fold。失败请求本身不重放,维持不重复生成的既有立场。/healthz 新增 generation 与活跃计数字段,传输失败日志附带脱敏的池快照与代理路由摘要(仅 scheme/host/port,绝不含凭证)。感谢 @Anionex 的故障取证(#12)与修复(#13)。