Summary
In the streaming chat handler, reservation.commit_tokens(0) is called immediately after the upstream stream is initiated, and the response is built with prompt_tokens: None, completion_tokens: None, total_tokens: None. The current code's own comment acknowledges this: "Streaming path doesn't compute cost yet (no token totals until the stream completes upstream). Phase 2 wires…"
Net effect: TPM caps and per-request cost telemetry are silently disabled for all streaming traffic. In production, streaming is the majority of LLM traffic.
Severity
🚨 HIGH — TPM enforcement and cost reporting are blind for streaming. Customers see "everything green" even when they've burned through their budget.
Location
crates/aisix-proxy/src/chat.rs:383 (verified on origin/main):
let upstream = bridge
.chat_stream(req, &ctx)
.await
.map_err(|e| with_model(ProxyError::Bridge(e)))?;
reservation.commit_tokens(0); // ← streaming path commits ZERO tokens
let sse_stream = build_sse_stream(upstream, now);
let response = Sse::new(sse_stream)
.keep_alive(KeepAlive::new().interval(Duration::from_secs(15)));
return Ok(Success {
response: response.into_response(),
provider: format!("{provider:?}").to_lowercase(),
model_id: model_id.clone(),
prompt_tokens: None,
completion_tokens: None,
total_tokens: None,
// Streaming path doesn't compute cost yet (no token totals
// until the stream completes upstream). Phase 2 wires…For comparison, non-streaming path (chat.rs:621) is correct:
reservation.commit_tokens(total); // ← actual token count
And cache-hit path (chat.rs:472) is correct by design:
reservation.commit_tokens(0); // cache hit — by design, no upstream tokens consumed
So 2 of 3 paths in the same function are right. Only the streaming success path is broken.
Why this is risky
- TPM cap not enforced for streaming: a customer with TPM=10k can run continuous streaming and the limiter never sees it
- Cost reporting shows $0 for streaming: dashboard cost graphs miss the majority of usage
- Cache-hit-saved metrics are also 0: because total tokens is 0, "saved tokens" derivation is also 0
- Stripe billing under-charges: if Stripe charges based on Cloud-side aggregated cost, customers underpay for streaming
Recommended fix
Wire the SSE stream completion to commit actual tokens:
let upstream = bridge.chat_stream(req, &ctx).await...?;
let sse_stream = build_sse_stream_with_token_callback(upstream, now, {
let reservation = reservation.clone();
move |total_tokens| {
reservation.commit_tokens(total_tokens);
}
});build_sse_stream already parses each SSE event — at the final data: [DONE] event (or end-of-stream), invoke the callback with the accumulated usage.total_tokens from the upstream's final usage frame.
For OpenAI: usage info comes in the last SSE event when stream_options: {include_usage: true} is set (which the gateway already sets unconditionally — see related issue #TBD on Anthropic Responses API breakage).
For Anthropic: message_delta events carry usage; final message_stop has the totals.
For providers that don't include usage in stream: estimate from completion tokens by tokenizer (cheaper than a non-streaming follow-up call).
Test case
test_streaming_chat_commits_actual_total_tokens_at_stream_end
test_streaming_chat_emits_nonzero_cost_telemetry
test_streaming_tpm_cap_enforced_after_total_tokens_exceed_limit
test_streaming_partial_response_on_client_disconnect_commits_partial_tokens
Source
Static scan, May 2026 (verified on origin/main:crates/aisix-proxy/src/chat.rs:383). Phase 2 work explicitly tagged in code comment.
Summary
In the streaming chat handler,
reservation.commit_tokens(0)is called immediately after the upstream stream is initiated, and the response is built withprompt_tokens: None, completion_tokens: None, total_tokens: None. The current code's own comment acknowledges this: "Streaming path doesn't compute cost yet (no token totals until the stream completes upstream). Phase 2 wires…"Net effect: TPM caps and per-request cost telemetry are silently disabled for all streaming traffic. In production, streaming is the majority of LLM traffic.
Severity
🚨 HIGH — TPM enforcement and cost reporting are blind for streaming. Customers see "everything green" even when they've burned through their budget.
Location
crates/aisix-proxy/src/chat.rs:383(verified onorigin/main):For comparison, non-streaming path (
chat.rs:621) is correct:And cache-hit path (
chat.rs:472) is correct by design:So 2 of 3 paths in the same function are right. Only the streaming success path is broken.
Why this is risky
Recommended fix
Wire the SSE stream completion to commit actual tokens:
build_sse_streamalready parses each SSE event — at the finaldata: [DONE]event (or end-of-stream), invoke the callback with the accumulatedusage.total_tokensfrom the upstream's final usage frame.For OpenAI: usage info comes in the last SSE event when
stream_options: {include_usage: true}is set (which the gateway already sets unconditionally — see related issue #TBD on Anthropic Responses API breakage).For Anthropic:
message_deltaevents carry usage; finalmessage_stophas the totals.For providers that don't include usage in stream: estimate from completion tokens by tokenizer (cheaper than a non-streaming follow-up call).
Test case
test_streaming_chat_commits_actual_total_tokens_at_stream_endtest_streaming_chat_emits_nonzero_cost_telemetrytest_streaming_tpm_cap_enforced_after_total_tokens_exceed_limittest_streaming_partial_response_on_client_disconnect_commits_partial_tokensSource
Static scan, May 2026 (verified on
origin/main:crates/aisix-proxy/src/chat.rs:383). Phase 2 work explicitly tagged in code comment.