Tower SSE thundering herd causes ephemeral port exhaustion on localhost

[amustafa]

Problem

The Tower server SSE endpoint (GET /api/events, port 4100) has a hard client cap of 50 (SSE_MAX_CLIENTS in tower-server.ts). When exceeded, the server evicts the oldest client to make room. The evicted client auto-reconnects, which pushes the count back to the cap, evicting another client — creating a thundering herd loop at up to ~800 connections/second.

Each short-lived TCP connection leaves a socket in TIME_WAIT for ~60 seconds. At peak churn this accumulates 2,900+ TIME_WAIT sockets against 127.0.0.1:4100, consuming ~10% of the ephemeral port range (32768–60999). This causes intermittent EADDRNOTAVAIL / connection-refused errors for all localhost services — not just Tower.

Root Cause Chain

1. SSE client count exceeds cap (50)
2. Server evicts oldest client (sseClients.shift() + res.end())
3. Evicted client reconnects (browser EventSource ~3s, VSCode via backoffDelayMs)
4. Reconnect pushes count back to cap → another eviction → cascade
5. Each connection leaves TIME_WAIT socket (~60s TTL)
6. Ephemeral port pool (28,232 ports) depleted under sustained load
7. Other localhost services (gRPC, Postgres, Kafka, Redis) fail intermittently

Evidence

Metric	Value
Peak connect rate	49,524 connects/min
TIME_WAIT sockets at diagnosis	2,888 against 127.0.0.1:4100
Port utilization at peak	~10% sustained, higher in bursts
Log file size (2.5 days)	1.18M lines, 774K SSE connect/disconnect

Code Locations

• Cap + eviction logic: packages/codev/src/agent-farm/servers/tower-server.ts — addSseClient
• SSE handler: packages/codev/src/agent-farm/servers/tower-routes.ts — handleSSEEvents
• Age-based eviction (heartbeat): packages/codev/src/agent-farm/servers/tower-server.ts — heartbeat interval
• SSEClient type: packages/codev/src/agent-farm/servers/tower-types.ts

Proposed Fixes (priority order)

1. Reject instead of evict (critical)

Return 503 + Retry-After when at capacity instead of evicting oldest. Rejection is a dead end; eviction is a chain reaction.

2. Raise cap from 50 to 200

SSE connections are lightweight. 50 is too low for workstations with multiple builders + dashboard tabs + VSCode.

3. Send retry: SSE directive

retry: 5000\n\n spaces out browser reconnections.

4. Throttle SSE connect/disconnect logging

774K log lines in 2.5 days. Heartbeat every 30s is sufficient visibility.

5. Stagger max-age eviction with per-client jitter

Fixed 5-min max-age causes synchronized eviction bursts. Add ±1 min jitter (4–6 min range).

[amustafa]

On it! Working on a fix now.