I've been using livekit for the past couple of weeks and have noticed a strange behavior: Sometimes the model will repeat a message (or something similar to it) after stating it.
When I started to investigate (using Portkey to evaluate traces), I realized that whenever a text message gets sent along-side a tool call, livekit will not register the text message in the chat context for the next round, causing the model to, sometimes, repeat the same message, given that it don't see it in its context.
Here is an example of what I saying, the image below is a message generated by GPT-4o in a STT-LLM-TTS framework:
On the next turn, that same message only have the tool call registered to it:
That causes problems on how the model generated the message for this turn.
I am using Python 3.12, livekit-agents and livekit-plugins-openai version 1.2.8, as well as portkey-ai==1.14.0
I believe the error is likely located within the _pipeline_reply_task for the Agent class, since I observed this for multiple models and with or without portkey (though I did not collect the evidence to add here)
I've been using livekit for the past couple of weeks and have noticed a strange behavior: Sometimes the model will repeat a message (or something similar to it) after stating it.
When I started to investigate (using Portkey to evaluate traces), I realized that whenever a text message gets sent along-side a tool call, livekit will not register the text message in the chat context for the next round, causing the model to, sometimes, repeat the same message, given that it don't see it in its context.
Here is an example of what I saying, the image below is a message generated by GPT-4o in a STT-LLM-TTS framework:
On the next turn, that same message only have the tool call registered to it:
That causes problems on how the model generated the message for this turn.
I am using Python 3.12, livekit-agents and livekit-plugins-openai version 1.2.8, as well as portkey-ai==1.14.0
I believe the error is likely located within the _pipeline_reply_task for the Agent class, since I observed this for multiple models and with or without portkey (though I did not collect the evidence to add here)