Skip to content

v1.8.2 — one abort listener per stream instead of one per chunk

Choose a tag to compare

@latentharbor latentharbor released this 02 Aug 10:07
· 16 commits to main since this release

DeadlineController.race added and removed an abort listener on every call, and it is called once
per body chunk. Invisible across the few dozen chunks of a 4 MB response; pure waste on an SSE
stream, which is one chunk per output token. The signal cannot change over the controller's
lifetime, so one registration is enough.

Measured in one isolate, 20k against 80k iterations differenced: 2.53 µs → 0.90 µs per call, −64%
(the floor, with no deadline machinery at all, is 0.23 µs).

This does not solve the cost of streaming. The gap to the platform's own fetch on an SSE
stream is ~155 µs per event; this is 1.6 of it, about 209 ms out of ~35 s on a 128,000-event
completion. It ships because re-registering a listener that cannot change is a waste that should not
have been there, not because it moves a bill.

A streaming-cost section, with two claims deliberately absent

Measured against a real streaming endpoint through a proxy, using the API's own usage block as the
token count rather than an estimate from response size: 250–310 µs per event against ~105 µs for
the platform's fetch. Events map to output tokens roughly 1:1, and because CPU and the model's
output charge both scale with output tokens, the ratio is constant — ~0.9% of the model bill at
any length.

Two claims from an earlier draft are absent because they did not survive re-measurement:

  • "Cost is flat in output length" was an artifact. max_completion_tokens never bound with the
    prompt used, so 512, 2000 and 8000 returned an identical 43 events and 11,834 bytes — three
    budgets measuring the same workload.
  • "HTTP/1.1 saves 13%" did not reproduce; h1 led one sweep and trailed the next, inside a ~20%
    run-to-run spread.

The section states that spread and gives a range rather than a figure.

1251 offline tests pass.