fix(client): TTL'd connection pool + resource-leak hardening#158
Merged
Conversation
Investigation started from Netty PrematureChannelClosureException warnings
("channel gone inactive with 1 missing response(s)") in the server logs, then
swept for the same bug class (silently-inherited library defaults / unreclaimed
resources on the long-lived server, whose window Neon scale-to-zero widens).
- HttpClientLayer: zio-http's default connection pool is Fixed(10) with no idle
TTL, so it keeps handing out keep-alive connections that Chess.com's Cloudflare
edge already reaped server-side; the next request fails mid-flight. Switch to a
Dynamic pool with a 30s TTL (< the edge idle timeout) so stale connections are
evicted before reuse. The connection-error retry stays the backstop.
- JobRunner: FileSink.make acquires a held-open fd whose only close path is wired
into the forked child's `ensuring`. An interrupt between the open and the fork
(the DB read in between can block seconds on a Neon cold start) leaked the fd.
Open the sink, register the promise, and fork all inside one uninterruptibleMask.
- CcasServer: the server inherited Server.Config.default's idleTimeout=None, so a
keep-alive client that vanished without FIN/RST pinned its channel + fd forever.
Set idleTimeout(60s), above JobLogStream.KeepAliveInterval so log-follow streams
aren't reaped.
- ChessComClient: each rate-limit throttle-down registered a new finalizer on the
layer-lifetime scope, accumulating one closure + Fiber shell per event for the
whole process lifetime. Hold the single recovery fiber in a Ref and
interrupt-and-replace, with one shutdown finalizer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Connection-pool and resource-lifecycle hardening, triggered by Netty
PrematureChannelClosureExceptionwarnings in the server logs and a follow-upaudit for the same bug class (silently-inherited library defaults / resources
never reclaimed on the long-lived server, whose window Neon scale-to-zero widens).
Fixed(10)connection pool(no idle TTL) with
Dynamic(0, 32, ttl = 30s). TheFixedpool kept handingout keep-alive connections that Chess.com's Cloudflare edge had already reaped
server-side, so the next request failed mid-flight (the source of the log
warnings). A TTL shorter than the edge idle timeout evicts stale connections
before reuse; the connection-error retry stays the backstop.
FileSink.makeopened aheld-open writer whose only close path was wired into the forked child, but the
interruptible gap before the fork (a DB read that can block on a Neon cold
start) could leak it. Open sink + register promise + fork now sit inside one
uninterruptibleMask.idleTimeout(60s)(default wasNone), so keep-aliveclients that vanish without FIN/RST can't pin a Netty channel + fd forever.
Chosen above
JobLogStream.KeepAliveInterval(20s) so log-follow streams aren'treaped.
rate-limit throttle-down; hold the single recovery fiber in a
Refandinterrupt-and-replace, with one shutdown finalizer.
Fixes
No issue numbers — found via log review + audit this session.
Testing
sbt Test/compileclean.sbt testgreen via the pre-push hook.TestChessComClientThrottling(incl. "recovery fibers are interrupted whenscope closes") and
TestJobRunnerexercised directly.🤖 Generated with Claude Code