Timeout parameter not working when using stream request

[shuhaojie]

I use httpx to make request to my LLM. Here is the code.

import httpx
import logging
import json

from httpx import TimeoutException


try:
   content = ''
   logging.info(f"request payload:{payload}")
   async with httpx.AsyncClient(timeout=LLM_TIMEOUT) as client:
      async with client.stream("POST", MODEL_URL, headers=HEADERS, json=payload,
                               timeout=LLM_TIMEOUT) as response:
          if response.status_code != 200:
              ans = await response.aread()
              logging.error(f"response.aread() error")
              raise Exception(f"Failed to generate completion stream: {ans}")
          async for line in response.aiter_lines():
              if line.startswith("data:"):
                  data_str = line[5:]
                  if data_str.strip() == "[DONE]":
                      break
                  try:
                      chunk_data = json.loads(data_str)
                      if chunk_data['choices']:
                          content += chunk_data["choices"][0]["delta"].get("content")
                  except json.JSONDecodeError:
                      logging.error("json parse error:", data_str)
      logging.info(f"request response:{content}")
except TimeoutException as te:
   logging.warning("request timeout")

Here the LLM_TIMEOUT is 60. However, I can see from my log that there exists some requests that costs more than 60 seconds.

[ehsandeep]

@shuhaojie -> https://github.com/encode/httpx

[dogancanbakir]

This repo is ProjectDiscovery's httpx, a Go HTTP toolkit. Your code uses the Python library of the same name; please ask at https://github.com/encode/httpx. Closing here.