Timeout with GPT-4, stream = true required

[Pierquinto]

It seems that with GPT-4 it takes too long to receive a response from APIs. In the reference they mention this stream = true to start receiving the first tokens immediately and avoiding a timeout.

[gehrisandro]

Hi @Pierquinto

I am able to use GPT-4 without any problems. Can you give us a concrete example when you run into the timeout issue?

After our last release in the past days we are now able to continue making the client capable of streamed responses. I will give you an update as soon as possible.

[afedukov]

I am also keep getting a lot of "Idle timeout reached" for model gpt-3.5-turbo chat requests:

[2023-03-20 08:38:13] prod.INFO: [ERROR] [openai_50] [217/500] Vocalex English [Idle timeout reached for "https://api.openai.com/v1/chat/completions".]  
[2023-03-20 08:38:16] prod.INFO: [Success] [openai_50] [189/500] Span-Gles [32 / 61]  
[2023-03-20 08:38:17] prod.INFO: [Success] [openai_500] [29/150] efWiki [250 / 304]  
[2023-03-20 08:38:39] prod.INFO: [ERROR] [openai_50] [481/500] TX Text Control ActiveX Server [Idle timeout reached for "https://api.openai.com/v1/chat/completions".]  
[2023-03-20 08:38:42] prod.INFO: [Success] [openai_50] [365/500] TX Text Control .NET Server [28 / 73]  
[2023-03-20 08:38:43] prod.INFO: [Success] [openai_50] [366/500] TX Text Control ActiveX Standard [17 / 52]  
[2023-03-20 08:38:47] prod.INFO: [Success] [openai_50] [367/500] Mpeg Splitter [55 / 134]  
[2023-03-20 08:39:16] prod.INFO: [ERROR] [openai_50] [219/500] Solar Eclipse Calculator [Idle timeout reached for "https://api.openai.com/v1/chat/completions".]  
[2023-03-20 08:39:17] prod.INFO: [ERROR] [openai_500] [30/150] SILEX [Idle timeout reached for "https://api.openai.com/v1/chat/completions".]  
[2023-03-20 08:39:17] prod.INFO: [Success] [openai_50] [190/500] Toy Train Widget [19 / 59]  
[2023-03-20 08:39:19] prod.INFO: [Success] [openai_50] [191/500] Gotham Search [30 / 59]  
[2023-03-20 08:39:23] prod.INFO: [Success] [openai_50] [192/500] Alpha Phi Alpha [31 / 80]  
[2023-03-20 08:39:25] prod.INFO: [Success] [openai_50] [193/500] Black Friday Countdown [31 / 88]  
[2023-03-20 08:39:27] prod.INFO: [Success] [openai_50] [194/500] EuroOffice Map Chart [38 / 88]  
[2023-03-20 08:39:30] prod.INFO: [Success] [openai_50] [195/500] FinanceJ [36 / 87]  
[2023-03-20 08:39:33] prod.INFO: [Success] [openai_50] [196/500] Kate Mara Screen Saver [39 / 89]  
[2023-03-20 08:39:35] prod.INFO: [Success] [openai_50] [197/500] IQ Geography Quiz [26 / 88]  
[2023-03-20 08:39:47] prod.INFO: [ERROR] [openai_50] [485/500] Stellar Phoenix Windows Data Recovery Software [Idle timeout reached for "https://api.openai.com/v1/chat/completions".]  
[2023-03-20 08:39:52] prod.INFO: [Success] [openai_500] [30/150] Drobo Dashboard [648 / 1412]  
[2023-03-20 08:40:09] prod.INFO: [Success] [openai_500] [31/150] Touchtip [278 / 331]  
[2023-03-20 08:40:21] prod.INFO: [Success] [openai_500] [32/150] BrainXT [213 / 327]  
[2023-03-20 08:40:32] prod.INFO: [Success] [openai_500] [33/150] RiftVault [191 / 226]  
[2023-03-20 08:40:35] prod.INFO: [ERROR] [openai_50] [228/500] Lock My PC [Idle timeout reached for "https://api.openai.com/v1/chat/completions".]  
[2023-03-20 08:40:39] prod.INFO: [Success] [openai_50] [198/500] python-slownie [63 / 143]  
[2023-03-20 08:40:42] prod.INFO: [Success] [openai_50] [199/500] Property Set Library [47 / 106]  
[2023-03-20 08:40:45] prod.INFO: [Success] [openai_50] [200/500] gapbuffer [44 / 68]  
[2023-03-20 08:40:47] prod.INFO: [Success] [openai_50] [201/500] filelike [17 / 45]  
[2023-03-20 08:40:47] prod.INFO: [ERROR] [openai_50] [486/500] CoffeeCup Free DHTML Menu Builder [Idle timeout reached for "https://api.openai.com/v1/chat/completions".]

[kingozorg]

change your php.ini and apache settings.

[ClaraLeigh]

How did y'all get access to the GPT 4 api already 😢

[jhull]

change your php.ini and apache settings.

Are you able to offer examples of this? I too have been having issues streaming GPT-4. I can do it with 3.5 but 4 keeps timing out (I keep hitting the 30s timeout on AWS (Vapor)).

I know some of the timeouts with 3.5 over the weekend were network related as they have been having many issues, but I would love to know how to properly set this up in production.

Locally, it streams fine--but obviously this is less than ideal for paying users.

[kingozorg]

I don't know about AWS. But i needed to set additional nginx directives on my server:

proxy_connect_timeout 360s;
proxy_send_timeout 360s;
proxy_read_timeout 360s;
fastcgi_send_timeout 360s;
fastcgi_read_timeout 360s;

Maybe that helps ... Cheers!

[gehrisandro]

Hi @Pierquinto

Hope our release yesterday resolved your issues with GPT-4.

Otherwise don't hesitate to reopen the issue.

[kmahmood74]

we are seeing lots of timeouts even though our timeout is set to 8 mins! This is with GPT-4. GPT3.5 works fine but has such a small token limit that it doesn't work for our use case

[dnnp2011]

I'm still seeing this pop up in the latest version (0.7.3), specifically for the GPT-4 model. It occurs exactly at 30 seconds, so it's clearly some kind of timeout. But strange that this doesn't also happen for GPT-3.5, only for GPT-4 model.

But I've already set every Nginx and PHP setting that I know of:

Nginx:

proxy_read_timeout 360s;
proxy_connect_timeout 20s;
proxy_send_timeout 360s;
client_body_timeout 360s;

fastcgi_buffers 16 32k;
fastcgi_buffer_size 64k;
fastcgi_busy_buffers_size 64k;
fastcgi_connect_timeout 360s;
fastcgi_send_timeout 360s;
fastcgi_read_timeout 360s;

PHP:

request_terminate_timeout = 180

[snake302]

the same for me