Skip to content

Handle retry-after HTTP header (Groq API) #3265

Description

@Pipeliner

Issue

Groq API has various limits, including tokens per minute: https://console.groq.com/docs/rate-limits.
When Aider encounter a 429 error, it does not use this header and implements its own delay scheme which does not work well.

Example:
Retrying in 1.0 seconds... 17:02:18 [24/1899]
litellm.RateLimitError: RateLimitError: GroqException - b'{"error":{"message":"Rate limit reached for
model gemma2-9b-it in organization ... service tier on_demand on
tokens per minute (TPM): Limit 15000, Used 11972, Requested 4351. Please try again in 5.289s. Visit
https://console.groq.com/docs/rate-limits for more
information.","type":"tokens","code":"rate_limit_exceeded"}}\n'
The API provider has rate limited you. Try again later or check your quotas.
Retrying in 2.0 seconds...
litellm.RateLimitError: RateLimitError: GroqException - b'{"error":{"message":"Rate limit reached for
model gemma2-9b-it in organization ... service tier on_demand on
tokens per minute (TPM): Limit 15000, Used 11390, Requested 4351. Please try again in 2.961s. Visit
https://console.groq.com/docs/rate-limits for more
information.","type":"tokens","code":"rate_limit_exceeded"}}\n'
The API provider has rate limited you. Try again later or check your quotas.
Retrying in 4.0 seconds...

Version and model info

v0.74.3.dev26+g185ea716
groq/gemma2-9b-it

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions