bug: Unable to capture output for openai stream=True

[moustafa-a]

Describe the bug

Hi,

I'm using the drop in replacement for openai. I am running into separate issues when stream = True

1. Iterating over the response throws an exception at the end
2. The output isn't captured in the trace that gets logged in the cloud

Neither of these issue are present when stream=False.

To reproduce

More details below.

Issue 1:

Running the following code throws an exception at the end

from langfuse.openai import OpenAI, openai

client = OpenAI()


response = client.completions.create(
    prompt = "tell me a joke",
    stream=True,
)

for chunk in response:
    print(chunk)

Exception

Traceback (most recent call last):
    for chunk in response:
  File ".venv/lib/python3.10/site-packages/langfuse/openai.py", line 864, in __iter__
    self._finalize()
  File ".venv/lib/python3.10/site-packages/langfuse/openai.py", line 888, in _finalize
    model, completion, usage = _extract_streamed_openai_response(
  File ".venv/lib/python3.10/site-packages/langfuse/openai.py", line 559, in _extract_streamed_openai_response
    completion += choice.get("text", None)

I was able to fix this by using changing the iterator to use the inner response object

for chunk in response.response:
    print(chunk)

Issue 2:

from langfuse.openai import OpenAI, openai

client = OpenAI()

openai.langfuse_debug = True

response = client.completions.create(
    prompt = "tell me a joke",
    stream=True,
)

for chunk in response.response:
    print(chunk)

The code runs but when I look at the trace online, the output is null

I tried adding
langfuse_context.flush() and openai.flush_langfuse() at the end as per the documentation but neither seem to do anything.

With the debugger on, I can see that when stream=False, there's a second call to update the trace. However, with stream=True that call isn't there.

DEBUG:langfuse:uploading data: {'batch': [{'id': 'ae6422a2-03af-4150-9166-19cd5980ae24', 'type': 'generation-update', 'body': {'traceId': 'd5d662c9-d499-4742-a911-8c334540057e', 'output': "Sure, here's a classic for you:\n\nWhy don't scientists trust atoms?\n\nBecause they make up everything!", 'id': '1b009061-6e47-4b55-b017-e3889060cff5', 'endTime': datetime.datetime(2025, 2, 11, 21, 35, 8, 276151, tzinfo=datetime.timezone.utc), 'model': 'gpt-4o-2024-05-13', 'usage': {'input': 11, 'output': 20, 'total': 31, 'unit': <ModelUsageUnit.TOKENS: 'TOKENS'>}, 'usageDetails': {'prompt_tokens': 11, 'completion_tokens': 20, 'total_tokens': 31, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}}}, 'timestamp': datetime.datetime(2025, 2, 11, 21, 35, 8, 277613, tzinfo=datetime.timezone.utc)}, {'id': '77c0abbe-b299-40f2-beef-a453db6487ed', 'type': 'trace-create', 'body': {'id': 'd5d662c9-d499-4742-a911-8c334540057e', 'timestamp': datetime.datetime(2025, 2, 11, 21, 35, 8, 277652, tzinfo=datetime.timezone.utc), 'output': "Sure, here's a classic for you:\n\nWhy don't scientists trust atoms?\n\nBecause they make up everything!"}, 'timestamp': datetime.datetime(2025, 2, 11, 21, 35, 8, 277729, tzinfo=datetime.timezone.utc)}], 'metadata': {'batch_size': 2, 'sdk_integration': 'openai', 'sdk_name': 'python', 'sdk_version': '2.57.13a0'}}

SDK and container versions

langfuse==2.59.2
openai==1.61.1

Additional information

No response

Are you interested to contribute a fix for this bug?

No

[dosubot]

I found a similar unsolved discussion titled "Not able to capture output for openai stream=True" which might be related to the issues you're experiencing ^[1].

To address the issues you're facing with the OpenAI drop-in replacement when using stream=True, you can try the following solution:

1. Exception at the end of iteration: Ensure that your application properly handles the final chunk with an empty choices list. OpenAI returns token usage information in a final chunk that has an empty choices list. You should check if the choices list is non-empty before accessing it:

   from langfuse.openai import openai
   
   client = openai.OpenAI()
   
   stream = client.chat.completions.create(
     model="gpt-4o",
     messages=[{"role": "user", "content": "How are you?"}],
     stream=True,
     stream_options={"include_usage": True},
   )
   
   result = ""
   
   for chunk in stream:
     # Check if chunk choices are not empty. OpenAI returns token usage in a final chunk with an empty choices list.
     if chunk.choices:
       result += chunk.choices[0].delta.content or ""
   
   openai.flush_langfuse()

   This code snippet ensures that you do not attempt to access an index in the choices list without checking if it is non-empty, thus preventing exceptions at the end of iteration ^[2].

2. Output not captured in the cloud trace: Using openai.flush_langfuse() ensures that all events are flushed and captured in the cloud trace. Make sure this function is called after processing the stream to ensure all data is sent to the Langfuse backend ^[2].

These steps should help resolve the issues you're encountering with stream=True.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[moustafa-a]

@dosu this doesn't address the problem

[dosubot]

To address the issue of not capturing output in the cloud trace when using stream=True, ensure you are using langfuse-vercel version 3.32.3 or later. Additionally, make sure you are using the Langfuse drop-in replacement for the OpenAI Python SDK by changing your import statement to:

from langfuse.openai import openai

Ensure your environment variables are set up with Langfuse credentials. When handling streaming responses, manage the final chunk with an empty choices list properly to ensure accurate token usage tracking. You can enable token usage tracking by setting {"include_usage": True} in the stream_options argument. Here's an example:

from langfuse.openai import openai

client = openai.OpenAI()

stream = client.chat.completions.create(
  model="gpt-4o",
  messages=[{"role": "user", "content": "How are you?"}],
  stream=True,
  stream_options={"include_usage": True},
)

result = ""

for chunk in stream:
  if chunk.choices:
    result += chunk.choices[0].delta.content or ""

openai.flush_langfuse()

This setup should help capture the output in the cloud trace effectively. Also, ensure your OpenAI SDK version is >=0.27.8 and that you have installed the Langfuse SDK with pip install langfuse openai. If issues persist, enable debug mode with openai.langfuse_debug = True to get more information about requests and responses ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[hassiebp]

Thanks for raising this, we will take a look!

[hassiebp]

@moustafa-a Could you please provide a full repro example? I cannot reproduce this with openai 1.64.0 and using this code:

from langfuse.openai import OpenAI, openai

client = OpenAI()


response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "tell me a joke"}],
    stream=True,
)

for chunk in response:
    print(chunk)

openai.flush_langfuse()

[moustafa-a]

I'm using the client.completions.create not client.chat.completions.create

from langfuse.openai import OpenAI, openai

client = OpenAI()


response = client.completions.create(
    prompt = "tell me a joke",
    stream=True,
)

for chunk in response:
    print(chunk)

[moustafa-a]

@hassiebp I was able to find the bug this line https://github.com/langfuse/langfuse-python/blob/3c78372169cf0185148982ffb0b6b8ec1cd56e09/langfuse/openai.py#L561 ->

  File ".venv/lib/python3.10/site-packages/langfuse/openai.py", line 938, in __aiter__
    await self._finalize()
  File ".venv/lib/python3.10/site-packages/langfuse/openai.py", line 962, in _finalize
    model, completion, usage = _extract_streamed_openai_response(
  File ".venv/lib/python3.10/site-packages/langfuse/openai.py", line 562, in _extract_streamed_openai_response
    completion += content
TypeError: can only concatenate str (not "NoneType") to str

changing it to:

            if resource.type == "completion":
                content = choice.get("text", None)
                if content:
                    completion += content

did the trick

@marcklingen any chance you guys can push this through, it's blocking us on production.

[hassiebp]

Thanks for debugging this. I'm curious when you see choice.text to not exist? I tried with a streamed completion (not chat), but couldn't reproduce.

from langfuse.openai import OpenAI, openai

client = OpenAI()


response = client.completions.create(
    model="gpt-3.5-turbo-instruct",
    prompt="tell me a joke",
    stream=True,
)

for chunk in response:
    print(chunk.choices[0].text, end="")

openai.flush_langfuse()

In any case, I've linked a PR that should fix this for you once released 👍🏾