Streaming breaks with OpenAI-compatible APIs that omit usage data

[stevegbrooks]

Problem

I'm using ellmer within my company (Boehringer Ingelheim). We have an OpenAI-compatible API service called Apollo. The output of Apollo's chat.completion when stream = TRUE doesn't include usage information in the streaming chunks. This causes ellmer to crash during token logging.

I've found that the error happens in tokens_row() when ellmer tries to create a data frame. Some fields have data but the usage fields are NULL, so you get mismatched row counts. Specifically, it throws this error: Error in data.frame: arguments imply differing number of rows: 1, 0.

Here's what a chat completion chunk looks like from Apollo when streaming (omitting the non-relevant parts):

{
  "choices": [{"delta": {"content": "Hello"}}],
  "usage": null
}

Compare this to OpenAI which sends usage data in the final chunk (this is also what our Apollo service looks like when giving non-streaming responses):

{
  "choices": [{"delta": {"content": ""}}],
  "usage": {"completion_tokens": 5, "prompt_tokens": 10, ...}
}

Steps to reproduce

Unfortunately, I can't give you a Boehringer Ingelheim account for you to test this with, so you're just gonna have to trust me :)

Expected behavior

This shouldn't break streaming. Either skip token logging when usage is missing, or use 0/NA as defaults.

Impact

This makes ellmer incompatible with our internal OpenAI-compatible service. The streaming content works fine - it's just the token logging that fails at the end.

Currently I have to catch this error and fall back to non-streaming mode, which defeats the purpose of streaming, and makes the usage of ellmer burdensome for users, especially when using tool calls and the wait time becomes longer.

Problematic code

I believe the issue occurs in /R/provider-openai.R at lines 272-277 in the value_turn() method:

cached_tokens <- result$usage$prompt_tokens_details$cached_tokens %||% 0
tokens <- tokens_log(
  provider,
  input = result$usage$prompt_tokens - cached_tokens,
  output = result$usage$completion_tokens,
  cached_input = cached_tokens
)

When result$usage is NULL (as with Apollo), attempting to access result$usage$prompt_tokens throws an error before the null-coalescing operator %||% and the existing null handling in tokens_log() can take effect.

Solution

I'm more than happy to submit the PR here, because I think I've found a good solution and its pretty easy to implement (at least for when the provider is "OpenAI"):

if (is.null(result$usage)) {
  tokens <- tokens_log(provider, input = NULL, output = NULL, cached_input = NULL)
} else {
  cached_tokens <- result$usage$prompt_tokens_details$cached_tokens %||% 0
  tokens <- tokens_log(
    provider,
    input = result$usage$prompt_tokens - cached_tokens,
    output = result$usage$completion_tokens,
    cached_input = cached_tokens
  )
}

When testing this, the symptoms I mentioned above go away, and I can happily use streaming w/ {ellmer} + Apollo.

The proposed fix would make the OpenAI provider more robust and consistent with how usage should be handled when it's missing from API responses. Not sure if there are any knock-on effects of this change, however. I imagine it would impact the token usage and cost reporting functionality, but since our service doesn't even provide token usage there's no way we can use it anyway.

[gadenbuie]

Thanks for the report! I think that's a reasonable fix, please go ahead and create the PR 😄

[stevegbrooks]

Ok I created it at #708

Thanks!

[schelhorn]

This is also a major issue for us (working at another phamaceutical, hi @stevegbrooks).

@gadenbuie: since the underlying issue is fixed now in #708 in DEV we would much appreciate a new CRAN release of {ellmer} soon so that we can continue working with our internal endpoints (which in our case are served by litellm proxy with another fancy name - the fix will thus likely help everyone who has a similar setup).

[By the way, @stevegbrooks, since you also may be using litellm: did you also experience an issue with custom {ellmer} tool functions that have no parameters erroring out with a NULL value exception? I am still trying to find out whether that is particular to our setup or a more general litellm issue, perhaps you could test that on your end? The test case would be to give the chat object a dummy tool function that has no parameters and ask the LLM to use it. Apologies for the slightly off topic.]

[stevegbrooks]

Hi @schelhorn ,

Nice to meet you!

We have a tool registered like this that takes no args:

'ellmer::tool(search_dataframes_tool, description = "...")'

and it works without error.

We can take this offline and discuss further. Email me at steven.brooks@boehringer.com!