ChatKit intermittently (but frequently) fails when responding to message with a file

[jacob-israel-turner]

To reproduce:

• As a user, upload and send a file
• The agent properly thinks
• The agent begins streaming a response
• Towards the end of a long response, often (~75% of the time in my case) the stream ends
• "There was an error while generating the assistant's response." UI is displayed

The console displays a QUIC error message:

Clicking "Retry" bricks the session.

Happy to provide any other details that might be helpful. Until this is resolved, my primary use-case for ChatKit does not work.

[jacob-israel-turner]

This error started showing up around the same time as #55 - unclear if they are related.

[weedon-openai]

Hey @jacob-israel-turner, definitely not related, we're working on #55 though.

Does this error happen with any file? Also, are you using Agent Builder or a custom ChatKit backend? Based on other issues, I assume Agent Builder?

[wilkes-oai]

Hi @jacob-israel-turner , thanks for reporting. You mentioned that it pretty consistently happens "towards the end of a long response." In terms of time duration, is that usually like 30 seconds, 1 minute, 2 minutes, 5 minutes? A ballpark here could be helpful as I analyze our chain of connections. Thank you!

[jacob-israel-turner]

Hey @jacob-israel-turner, definitely not related, we're working on #55 though.

Does this error happen with any file? Also, are you using Agent Builder or a custom ChatKit backend? Based on other issues, I assume Agent Builder?

I've only tested with PDFs - it fails much more consistently with bigger files. I think it's a combination of larger files and larger prompt. It pretty consistently fails with files >=250KB and a prompt >=1.8k tokens. Anything smaller than that tends to run successfully. The output is ~1.5k-2k tokens.

I am using Agent Builder - it's a very simple configuration:

Hi @jacob-israel-turner , thanks for reporting. You mentioned that it pretty consistently happens "towards the end of a long response." In terms of time duration, is that usually like 30 seconds, 1 minute, 2 minutes, 5 minutes? A ballpark here could be helpful as I analyze our chain of connections. Thank you!

In general the agent will think for 60-90 seconds, then generate a response for ~30 seconds before failing. Sometimes it fails while thinking though.

[wilkes-oai]

That's very helpful! I'm actually pushing a change right now that hopefully will help here, I'm extending some configured timeouts to be much longer. This should take effect in an hour or less from now.

[jacob-israel-turner]

Woo, looks like it's fixed! I've tried files up to 4.6MB without issue.

Thanks for your help @wilkes-oai and @weedon-openai. I'll re-open if this issue shows up again.

[wilkes-oai]

Great to hear, thanks!