Skip to content

freellmpool 0.9.2

Choose a tag to compare

@0xzr 0xzr released this 03 Jun 05:19

Performance: connection pooling.

  • Requests reuse a keep-alive httpx client instead of opening a fresh TCP+TLS connection each call — big latency win for repeated calls to the same provider (agent loops, the proxy). ~0.15s/call warm.
  • Fast-fail connect timeout (10s) so dead providers fail over quickly.
  • Quota ordering snapshots once per request instead of per-candidate locked reads (matters for the 300+ model catalog).

pip install -U freellmpool