How to get the usage of "CompletionResponseUsage" returned by OpenAI API?

[GeorgeGalway]

Describe the feature

stream mode currently does not include prompt_tokens, completion_tokens, and total_tokens. Would it be possible to add these features?

[transitive-bullshit]

@GeorgeGalway have you tried the detail property of the returned result? It should contain the unedited JSON response returned by the OpenAI API. If it doesn't exist, then that's a bug on our end.

[GeorgeGalway]

@GeorgeGalway have you tried the detail property of the returned result? It should contain the unedited JSON response returned by the OpenAI API. If it doesn't exist, then that's a bug on our end.

I'm using v5.0.11, stream mode, and detail property doesn't include prompt_tokens, completion_tokens, and total_tokens.

[transitive-bullshit]

Hmmm yeah the onProgress version won't have this info because the partial responses don't have the full token counts yet. We should still get this info right before the last [DONE] message, though (I think), so it may be a bug in the stream version that we're not returning this properly at the end.

[transitive-bullshit]

I just did a quick test w/ the stream version of OpenAI's chat completion endpoint, and I'm not seeing this info returned at any point once the stream is finished.

I may be looking in the wrong place, but I think we just don't get this info from OpenAI in the stream version.

[transitive-bullshit]

It's always a 'chat.completion.chunk' object which doesn't contain the usage info.

[GeorgeGalway]

It's always a 'chat.completion.chunk' object which doesn't contain the usage info.

Perhaps we can know the tokens that were sent?

[GeorgeGalway]

Hmmm yeah the onProgress version won't have this info because the partial responses don't have the full token counts yet. We should still get this info right before the last [DONE] message, though (I think), so it may be a bug in the stream version that we're not returning this properly at the end.

I can calculate the number of characters from the returned result and can also see the sent tokens in the log, but I cannot retrieve the number of sent tokens from the detail.

[transitive-bullshit]

Hmmm that'd be nice, yeah. There are _ methods that you can calculate this (_buildMessages and _getTokenCount).

It might be nice to expose this in a cleaner way, but I don't have time; PRs welcome :)

[JokerQyou]

According to the OpenAI cookbook, usage info is not available when stream mode is enabled.

Note that using stream=True in a production application makes it more difficult to moderate the content of the completions, as partial completions may be more difficult to evaluate. which has implications for approved usage.

Another small drawback of streaming responses is that the response no longer includes the usage field to tell you how many tokens were consumed. After receiving and combining all of the responses, you can calculate this yourself using tiktoken.

[alxmiron]

Guys, please take a look at my proposal: #507