fix(fetch): bound EDGAR fetches in flight, not just their start rate

[sroussey]

sec spac process <ciks...> exhausts the process's file-descriptor table once it leaves the already-downloaded issuers.

Cause

The fetch queue caps starts per second and nothing else. JobQueueWorker dispatches each claimed job in the background and immediately loops for the next, and the rate limiter's window is pruned by age rather than by completion, so a slot frees one second after a fetch begins no matter how long it runs.

In-flight work is therefore rate x latency. While EDGAR is sub-second that is ~8; a slow spell serving multi-MB full-submission .txt documents at 30s each admits ~240 concurrent requests. Each holds roughly two descriptors and the pool releases them only after an idle period, so the peak is what runs the table dry.

Measured on Bun 1.3.11, driving the real safeFetch against a slow local server:

concurrent requests	peak fds
10	38
40	98
100	218
200	418

At ~2 descriptors per in-flight request, macOS's default ulimit -n of 256 is crossed at about 128 concurrent fetches.

This is not a leak: at fixed concurrency the count is flat across repeated rounds (98 → 98 over 10 rounds) and returns to baseline after ~20s idle. Only the unbounded peak needed capping.

Cache-hit issuers never fetch, which is why a single already-downloaded issuer does not reproduce it.

Fix

Add a ConcurrencyLimiter to the queue's CompositeLimiter. Unlike the rate limiters it holds its token until the job reaches a terminal state, which is what makes it a concurrency bound rather than a second rate cap.

It sits ahead of the rate limiters: it is an in-process counter, so a claim it rejects under saturation costs nothing, whereas acquiring the cluster limiter first would spend a reserve/release round trip against Postgres on every rollback.

Default 16 via SEC_FETCH_MAX_CONCURRENT (clamped 1–64), so the cap binds only once a fetch averages over two seconds — a healthy sweep runs at exactly today's speed, and a degraded EDGAR costs throughput instead of the whole process.

The two limits are independent and both are needed. The rate limiter cannot be derived from the concurrency one (λ = C/W floats with the peer's latency: at C=16 against a 50ms response that is ~320 req/s, well past EDGAR's ceiling), and their scopes differ — the EDGAR quota is per-IP and cluster-shared, descriptors are per-process.

Behaviour is identical on both backends: PostgresRateLimiterStorage counts WHERE executed_at > now - windowMs, the same age-pruned window on starts as the in-memory path. CompositeLimiter.scope was already "process" (EvenlySpacedRateLimiter is process-scoped), so no start-up warning changes.

Testing

SecFetchConcurrency.test.ts drives the real queue against a slow server and asserts the peak never exceeds the cap, with a lower bound so a silently-serialized queue cannot pass vacuously. Verified it fails without the fix:

AssertionError: expected 4 to be less than or equal to 2

• bunx tsc --noEmit — clean
• src/task/fetch, src/task/spac, src/config — 169 passed, 9 skipped

Note

While measuring this I confirmed a separate, unrelated defect in libs: under Bun, undici's dispatcher option is ignored, so SafeFetch.server.ts's DNS-rebinding connection pin is a silent no-op on the runtime this CLI ships on. Filed as workglow-dev/libs#789 — it is a security issue, not a resource one, and does not affect this change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SUkk4LNNdcYUD2v1FMvkvL

---

Generated by Claude Code