429 Too Many Requests

[arnomoonens]

I am getting a 429 error from the OpenAI API when iterating over a model.pipe result.
I am using @llm_tasks = "spacy.NER.v2" and

[components.llm.model]
@llm_models = "spacy.GPT-3-5.v1"
name = "gpt-3.5-turbo"
config = {"temperature": 0.3}

This is the traceback:

  File "/workspaces/project/llm.py", line 16, in <genexpr>
    model_result_fin = (convert_llm_entities(doc) for doc in model_result_fin)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/spacy/language.py", line 1611, in pipe
    for doc in docs:
  File "/usr/local/lib/python3.11/site-packages/spacy/util.py", line 1705, in _pipe
    yield from proc.pipe(docs, **kwargs)
  File "spacy/pipeline/pipe.pyx", line 55, in pipe
  File "/usr/local/lib/python3.11/site-packages/spacy/util.py", line 1705, in _pipe
    yield from proc.pipe(docs, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/spacy_llm/pipeline/llm.py", line 175, in pipe
    error_handler(self._name, self, doc_batch, e)
  File "/usr/local/lib/python3.11/site-packages/spacy/util.py", line 1724, in raise_error
    raise e
  File "/usr/local/lib/python3.11/site-packages/spacy_llm/pipeline/llm.py", line 173, in pipe
    yield from iter(self._process_docs(doc_batch))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/spacy_llm/pipeline/llm.py", line 199, in _process_docs
    responses_iters = tee(self._model(prompts_iters[0]), n_iters)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/spacy_llm/models/rest/openai/model.py", line 115, in __call__
    responses = _request(
                ^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/spacy_llm/models/rest/openai/model.py", line 86, in _request
    r = self.retry(
        ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/spacy_llm/models/rest/base.py", line 147, in retry
    raise ConnectionError(
ConnectionError: API could not be reached after 32.592 seconds in total and attempting to connect 5 times. Check your network connection and the API's availability.
429     Too Many Requests

Could you please help with this?
Is there some kind of throttling mechanism that could resolve this?

[rmitsch]

Hi @arnomoonens, this looks like you hit OpenAI's API rate limit. There's not much we can do about that directly, but you can, as for all models, increase the max_tries and timeout parameters to better avoid hitting the rate limit (see docs).

[arnomoonens]

Thanks for the information.
However, the actual issue was that my account was somehow switched to prepaid billing and I didn't have any credit. When using openai package directly, I get the right error about exceeding my quota, but when using spacy-llm I get the 429 error. This is quite confusing I think...

[rmitsch]

Yeah, that's confusing. Can you let me know the exact error message you're getting with the openai package?

[arnomoonens]

RateLimitError: You exceeded your current quota, please check your plan and billing details.

[rmitsch]

Thanks! Added this to our backlog, we'll improve the error message for this.

[ishaan-jaff]

@arnomoonens @rmitsch

I'm the maintainer of LiteLLM we provide an Open source proxy for load balancing Azure + OpenAI + Any LiteLLM supported LLM
It can process (500+ requests/second)

From this thread it looks like you're running in 429 rate limit errors. Our proxy will allow you to maximize throughput by load balancing between Azure OpenAI instance - I hope our solution makes it easier for you. (i'd love feedback if you're trying to do this)

Here's the quick start:

Doc: https://docs.litellm.ai/docs/simple_proxy#load-balancing---multiple-instances-of-1-model

Step 1 Create a Config.yaml

model_list:
  - model_name: gpt-4
    litellm_params:
      model: azure/chatgpt-v-2
      api_base: https://openai-gpt-4-test-v-1.openai.azure.com/
      api_version: "2023-05-15"
      api_key: 
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4
      api_key: 
      api_base: https://openai-gpt-4-test-v-2.openai.azure.com/
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4
      api_key: 
      api_base: https://openai-gpt-4-test-v-2.openai.azure.com/

Step 2: Start the litellm proxy:

litellm --config /path/to/config.yaml

Step3 Make Request to LiteLLM proxy:

curl --location 'http://0.0.0.0:8000/chat/completions' \
--header 'Content-Type: application/json' \
--data ' {
      "model": "gpt-4",
      "messages": [
        {
          "role": "user",
          "content": "what llm are you"
        }
      ],
    }
'

[rmitsch]

Thanks for letting us know! We'll take it into consideration. Will also convert this into a discussion, as this is not a bug/feature request per se.