New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Hitting Finch Timeout #48
Comments
@gworkman for the time being i would prefer not to add configuration. I would rather set the default timeout to a high enough value that all normal use-cases are addressed. WDYT about doubling the default Finch timeout from 15s to 30s? Or I could even raise it to 60s. |
@gworkman I increased the default receive timeout to 45 seconds, and have published to hex.pm as v 0.2.1. can you check that this resolves your problem? |
Works for me, thank you! I will check today and let you know :) |
@gworkman great. please let me know if it works, so I can close this issue. |
Good news! Extending the timeout works great, it is always returning some response now. Bad news: the problem is with OpenAI. I have been getting these responses from the endpoint after calling %{"error" => %{"code" => nil, "message" => "That model is currently overloaded with other requests. You can retry your request, or contact us through our help center at help.openai.com if the error persists. (Please include the request ID in your message.)", "param" => nil, "type" => "server_error"}} Fortunately, I didn't really need to change my code - I have been running these calls in Happy to contribute if you give me some guidance! :) |
@gworkman I like the good news :-) For the moment, I'd like to keep the library as a very thin wrapper on the OpenAI REST API. So I'd prefer not to "fatten" the client by adding support at this level (at least not yet). What would be immediately useful would be sample code (hopefully as a Livebook) which shows how you're handling the error / retries, etc, so all users are aware of problematic scenarios and how to approach them. If you contribute a sample "Error Handling" livebook, i'd be happy to add it to the docs. |
@restlessronin I am seeing the Mint Timeout error using the gpt-4 turbo model. It's taking more than 2 minutes to respond when the token size is greater than 1500. With the token size being 128K for the latest models, this is an occurrence 7/10 times for my use case. Could you please make it configurable as suggested above? or Change the |
@SinKasula I have made the timeout configurable using I have also added documentation covering configuration to the user guide. Changes should already be available on hex as v0.5.2. |
@SinKasula can you confirm that the fix solved your problem? Thanks |
@restlessronin are you still against providing configuration for Finch? I would love to tweak some of the connection options too and I don't see a way to do it. Having a glance at the code, I think the way to do it would be to hook into this line: we could grab the options from either Config and/or specify a callback function in config that is called to fetch the configuration - I don't mind either solution. |
@restlessronin The above fix did not work. I think the options are not being set on the Finch request. |
@SinKasula My bad. I made an incorrect assumption about |
@hubertlepicki I moved your comment and my response to a new issue |
@restlessronin Thank you! This solved my problem |
Describe the bug
I'm getting the following timeout error while making requests to the Chat Completions endpoint.
Can we add some timeout options in the application config?
Finch.request!/3
supports the keyword options:pool_timeout
and:receive_timeout
.Note: this only happens about 5-10% of the time, and usually only with longer requests (1500+ tokens). Retrying the request usually completes within 1-2 seconds.
Also not sure if it is a factor, but I'm located in Europe and OpenAI only has US servers for now it seems
To Reproduce
Run (and/or add small variations) until it times out
Code snippets
No response
OS
Mac
Elixir version
1.14.5
Library version
v0.2.0
The text was updated successfully, but these errors were encountered: