Slow performance

[gastonmorixe]

Describe the bug

I am not sure why yet but our requests are taking like between 5 to 15 seconds most times.

I tried to log using httplog gem and it stops after the connecting step, after that for like 10 seconds goes fast.

May it be related to httparty gem using Net::HTTP? I think it doesn't support http2 nor keep-alives

[jtoy]

I see many requests time out but looking at the API status page, it shows up as fine. I also tested with python and the code takes a while to run, but it doesn't error with a timeout at least.

[jtoy]

FYI I see a new version came out to help address this issue: https://github.com/alexrudall/ruby-openai/releases/tag/v3.6.0

[gastonmorixe]

@jtoy v3.6 didn't change much. I did some research and tests and it seems to me that a big part of the slow performance is due to HTTParty's Net::HTTP usage.

Quickly hacking HTTParty I enabled keep-alive to persist the connection and it improved by around ~50%.

I think moving to another gem that doesn't rely on Net::HTTP and enabling streaming is key to unlock good performance.

[bf4]

I think the OpenAI API itself is unstable. But the idea of using net-http-persistent style keep-alive when available or via adapters sounds it could be a good approach as well. Probably part of #189

[gastonmorixe]

I agree

[deikka]

@jtoy v3.6 didn't change much. I did some research and tests and it seems to me that a big part of the slow performance is due to HTTParty's Net::HTTP usage.

Quickly hacking HTTParty I enabled keep-alive to persist the connection and it improved by around ~50%.

I think moving to another gem that doesn't rely on Net::HTTP and enabling streaming is key to unlock good performance.

How did you hacked HTTParty? Thanks.

[dep]

I think moving to another gem that doesn't rely on Net::HTTP and enabling streaming is key to unlock good performance.

Does such a gem even exist though?

[alexrudall]

#234 adds streaming with Faraday - final reviews much appreciated 👍

[panozzaj]

I looked and #234 is now merged. We may be able to close this issue if the performance issues mentioned here are now fixed with the change to Faraday. (I only note this as a new user of this library who was checking issues and saw this since performance seems important.)

[jtoy]

One more comment, I’m not sure which is better, it I’ve personally moved to http.rb for all networking requests. I’m not sure if that would handle it better. Also in my Python code I’ve written retry logic. I’m not sure if that would make sense to put in the Ruby libraries.