Skip to content

Fix: HTTP client connection pool leak causing agent failures#440

Merged
Zakaria-Kofiro merged 9 commits intomasterfrom
fix/connection-pool-leak
Dec 16, 2025
Merged

Fix: HTTP client connection pool leak causing agent failures#440
Zakaria-Kofiro merged 9 commits intomasterfrom
fix/connection-pool-leak

Conversation

@Zakaria-Kofiro
Copy link
Copy Markdown
Collaborator

@Zakaria-Kofiro Zakaria-Kofiro commented Nov 7, 2025

HTTP CLIENT CONNECTION LEAK FIX:

CHANGES:

  • Always consume HTTP response entity regardless of status code (fixes CLOSE_WAIT accumulation)
  • Add response.close() in finally block to ensure connection release
  • Set setConnectionManagerShared(false) to let HttpClient manage pool lifecycle
  • Add defensive BR/GZIP decompression with null/empty checks

ROOT CAUSE:
Response bodies weren't consumed for non-2xx status codes, leaving connections stuck in CLOSE_WAIT state. Adding BR support amplified the issue by throwing EOFException on empty bodies.

VALIDATION:
2+ hour A/B test at steady state (~19K concurrent connections)

  • Baseline: 1,400+ leaked CLOSE_WAIT, agents crash with too much load
  • Fix: CLOSE_WAIT cycles 2-110 (healthy), zero crashes
  • CPU/Memory: Fix uses same or less CPU/Memory (~10% less)
  • FDs: 50% reduction (22K → 8-12K)

Please make sure these check boxes are checked before submitting

  • ** Squashed Commits **
  • ** All Tests Passed ** - mvn clean test -P default

** PR review process **

  • Requires one +1 from a reviewer
  • Repository owners will merge your PR once it is approved.

@Zakaria-Kofiro Zakaria-Kofiro force-pushed the fix/connection-pool-leak branch from 0ab6b72 to 45f3417 Compare December 10, 2025 21:23
@Zakaria-Kofiro Zakaria-Kofiro marked this pull request as ready for review December 15, 2025 00:01
@Zakaria-Kofiro Zakaria-Kofiro merged commit 5d96a52 into master Dec 16, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants