Feature: Add OpenAI Usage stats when using streaming with the Chat Completions API or Completions API

[douxiaofeng99]

from OpenAI 1.26.0, Usage stats now available when using streaming with the Chat Completions API or Completions API

https://community.openai.com/t/usage-stats-now-available-when-using-streaming-with-the-chat-completions-api-or-completions-api/738156

FastChat should also support this stat.

[brandonbiggs]

For what it's worth, Fastchat already supports some usage stats while streaming:
Example from the fastchat openai api: "usage": {"prompt_tokens": 591, "total_tokens": 674, "completion_tokens": 83}

[douxiaofeng99]

Thanks for reply. Could you tell me which version support usage stats. Then, you can close this issue!

[brandonbiggs]

I'm using the newest version, but I think it's been available for a little bit. https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/openai_api_server.py#L743

[douxiaofeng99]

I reviewed the code and found its' for @app.post("/v1/embeddings", dependencies=[Depends(check_api_key)])
@app.post("/v1/engines/{model_name}/embeddings", dependencies=[Depends(check_api_key)]).
In the above section, I mean Chat Completions API or Completions API. We also download the latest package and sure that there is no usage stat when use streaming mode.

[douxiaofeng99]

@brandonbiggs any progress?

[brandonbiggs]

Sorry, any progress on what? I get stats when calling mine. Not sure why you don't.

[douxiaofeng99]

" I get stats when calling mine. Not sure why you don't." Do you use streaming mode(SSE)? If so, i will test again

[douxiaofeng99]

any one can help? in the https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/openai_api_server.py, the /v1/chat/completions method without stat response when use streaming.