Timeout was reached

[kalimu]

When the prompt and output are large and LLM needs more time to generate the response the $chat() method is prone to result in

! Failed to perform HTTP request.
! Timeout was reached [api.openai.com]: SSL/TLS connection timeout

It would be good to have some way to set / increase httr2::req_timeout() in chat_perform_value https://github.com/tidyverse/elmer/blob/47ef8ecbb562fa208ba59176794bdaf58273a429/R/httr2.R#L31

[kalimu]

This serves as a workaround:

custom_function <- function(provider, req) {
  req |> 
    httr2::req_timeout(60) |>
      httr2::req_perform() |> 
    httr2::resp_body_json()
  }
unlockBinding("chat_perform_value", asNamespace("elmer"))
assign("chat_perform_value", custom_function, envir = asNamespace("elmer"))
lockBinding("chat_perform_value", asNamespace("elmer"))

[hadley]

Do you have an example of what size prompt + what provider causes this problem?

[kalimu]

Do you have an example of what size prompt + what provider causes this problem?

I noticed this issue with OpenAI API. However, it is not easy to replicate as it has transient character, although it happens regularly.
I believe from time to time OpenAI is less responsive (because of heavy load or some other issues) hence the time out errors with default httr2 settings.
But it's not determined by the prompt or token numbers only - if the API performance is as expected the prompts don't matter, if the performance is worse prompt close to the context window can give timeout errors. If the API performance is really bad, even small prompts requests can time out.

But being able to set custom / increase timeout limits in httr2 usually solve those problems.

[schmidb]

I have the same issue with OpenAI API. It is happening very randomly. Not yet able to reproduce it.

[schmidb]

I don't think this really helps. But this is the error I am running into every 5th or 6h call of OpenAI.

Error in `req_perform()`:
! Failed to perform HTTP request.
Caused by error in `curl::curl_fetch_memory()`:
! Timeout was reached [api.openai.com]: SSL/TLS connection timeout
Backtrace:
     ▆
  1. ├─base::source("Rcode/GenAISummary.R")
  2. │ ├─base::withVisible(eval(ei, envir))
  3. │ └─base::eval(ei, envir)
  4. │   └─base::eval(ei, envir)
  5. └─chat$chat(ContentText(single_string), "Create a 5 sentence summary of the insights without describing features of the report or recommendations.\n  Add one more paragraph with 3 sentences how the Linkedin profiles performs compared to the benchmark data especially in view on the last 90 days. \n  Add one more paragraph with two hands-on tips how to improve the linkedIn content strategy.\n  Format the output as html and start with <h2> and the paragraphs with <p>. No sub headlines. No html body or header is required.")
  6.   └─coro::collect(...)
  7.     └─coro:::reduce_steps(x, steps, along_builder(list()))
  8.       └─coro:::reduce(x, reducer, .init = identity)
  9.         └─coro:::reduce_impl(.x, .f, ..., .init = .init)
 10.           └─coro:::iter_reduce_impl(.x, .f, ..., .left = .left)
 11.             ├─coro::is_exhausted(new <- .x())
 12.             └─coro (local) .x()
 13.               └─base::evalq(...)
 14.                 └─base::evalq(...)
 15.                   └─base::evalq(...)
 16.                     └─base::evalq(...)
 17.                       ├─coro::is_exhausted(elt <- iterator())
 18.                       └─coro (local) iterator()
 19.                         └─base::evalq(...)
 20.                           └─base::evalq(...)
 21.                             ├─base::evalq(...)
 22.                             │ └─base::evalq(...)
 23.                             │   └─coro (local) user(...)
 24.                             │     ├─.last_value <<- eval_bare(substitute(expr), user_env)
 25.                             │     │ └─rlang::env_poke(env, lhs, value, inherit = TRUE, create = FALSE)
 26.                             │     └─rlang::eval_bare(substitute(expr), user_env)
 27.                             └─elmer:::chat_perform(...)
 28.                               └─elmer:::chat_perform_value(provider, req)
 29.                                 ├─httr2::resp_body_json(req_perform(req))
 30.                                 │ └─httr2:::check_response(resp)
 31.                                 │   └─httr2:::is_response(resp)
 32.                                 └─httr2::req_perform(req)
 33.                                   └─base::tryCatch(...)
 34.                                     └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 35.                                       └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 36.                                         └─value[[3L]](cond)
Execution halted

[hadley]

It's weird that I've never seen this issue — I've seen a ton of other random problems because the elmer checks run pretty frequently, but not this one.

[CorradoLanera]

It is also weird that it never happen with official Python api. It is quite hard to replicate, but if you try long and complex context (e g. Instruction for the screening phase of a systematic review) and run it many times, maybe in parallel (e.g. on some hundreds or thousands of abstracts) it will happen quite always. Maybe if you repeat the query on the errored records only, sometimes you solved sometimes the error remains there. Anyway it never happen using Python api (not a single time in my experience, as I mentioned in irudnyts/openai#61 (comment)), which lead me to add an option for that in gpteasyr (https://github.com/CorradoLanera/gpteasyr?tab=readme-ov-file#pythons-backend). Hack timeouts (one or more of the many ones) sometimes solves the issue, but never consistently... sometimes the only solution I found was switch to the Python backend (e g , w/ reticulate), which btw always solves the issue immediately 🤔🤷

[hadley]

One hack would be to set req_retry(retry_on_failure = TRUE), but that's not going to give a great user experience if the connection takes 10 seconds to time out.

[hadley]

Oh is this all when echo = FALSE and you're not streaming? In that case, 10s might not be enough time for the mode to finish streaming (as hinted at in https://community.openai.com/t/frequently-getting-api-timeout-error-what-am-i-doing-wrong/611941/2) and we should bump up the timeout.

But the docs suggest that https://curl.se/libcurl/c/CURLOPT_CONNECTTIMEOUT.html only refers to the connection phase, which I think would have completed.

If someone could come up with a non-private reprex that allowed me to see the problem, it's likely I could fix it much faster.

[hadley]

I think the solution here is to just bump up the overall timeout for non-streaming queries.