Skip to content

Release b528

Choose a tag to compare

@github-actions github-actions released this 18 Sep 02:41

fix(metrics): stop counting protocol lifecycle events as the first token

TTFT was stamped on the very first SSE chunk. For /v1/responses the first chunk is
response.created, a protocol event the upstream emits as soon as the response object
exists, long before the model produces anything. The recorded value was therefore
'headers to first protocol event' (~1 ms) rather than time to first token.

Evidence: 48 of 52 responses rows in the reference DB sat under 5 ms (median
0.92 ms) while the same requests ran 60 ms to 320 s, and the TTFT/latency ratio
was ~0.004%. Replaying the smallest stored request straight against the upstream
gave response.created at 25 ms, response.in_progress at 26 ms, then
response.output_item.added at 1378 ms and the first reasoning delta at 1378 ms -
so the true first token was ~1378 ms while the gateway logged 0.076 ms.

Both streaming loggers now skip the lifecycle events when stamping TTFT:
response.created / response.in_progress / response.queued for the Responses API,
message_start / ping for Anthropic, which has the same shape (message_start is
emitted immediately). Chat is unaffected because its first chunk already carries
model output, which the reference data confirms (no chat row below 370 ms).

Verified by replaying the same request through the gateway against the real
upstream: the client saw the first output event at 1524 ms and the first delta at
1525 ms, and the gateway now records TtftMs = 1500 ms instead of 0.076 ms. Three
unit tests cover both skip lists.