Repository navigation
Free-tier Groq rate limits cause multi-hour runtime with no visible warning (parallel_attempts + low TPM) #2215
Replies: 5 comments
|
One useful refinement for the proposed warning: distinguish concurrency from token throughput. Reducing Groq documents For a small PR, a bounded, deduplicated message at the generator retry boundary seems useful: first 429 explains the cause; periodic summaries report retry count and wait duration; recovery clears the warning state. Don't print the full request/headers because they can contain credentials or sensitive prompts. A future shared limiter could pace all attempts against the same provider budget; independent exponential backoffs alone can wake up together and create another burst. I can't decide maintainer scope, but your log provides a good case for that observability improvement even without changing scheduling behavior. |
|
There are multiple recent enhancements either in recently in flight, landed, or reported that expose this as an area for UX improvement. In part this behavior is intended function as "garak is tenacious". An idea on adding a user configurable threshold has been floated in #2141, and comments in that issue offer some insight into why current behavior is desired as a default for common usage patterns. How to define and design the common knobs across generators for a threshold to utilize when evaluating health of the run is still somewhat up in the air though some mechanism for simply creating stdout or stderr warnings that there is a high retry rate for some window is likely a good first iteration of improving the user experience by providing feedback about how the default functions without changing behavior. It may be worth some time investigating if there are runtime metrics available from the backoff decorators that could aid in collecting details of the targets overall backoff rates. Note there may be some concurrency coordination issues as each Depending on if backoff metrics provide an ability to gather time bound metrics this pattern may not work well when parallelization is not requested as the rate of acceptable errors on the single process run differ if accumulation from the beginning of the run does not account for acceptable periods of high errors rates. Some probes can induce high timeout error rates that only occur for prompts it creates and prompts from subsequent probes should not trigger a failure for a single or low number of transient issues that happen to occur after such probe completes. |
|
Thanks both, this is really helpful context. @jmartin-tech, makes sense re: tenacious-by-default being intentional, and #2141 gives good background on why. Agreed a stdout/stderr warning that doesn't change default behavior is the right scope for a first pass, rather than trying to solve the configurable-threshold question up front. @identity-wael, the concurrency vs. token-throughput distinction is a good catch, --parallel_attempts alone wouldn't have explained what I saw if my prompts had been larger. I like the shape you proposed: first 429 explains the cause, periodic summaries report retry count + wait duration, recovery clears the state. Using the Retry-After/remaining/reset headers to name the exhausted limit (tokens vs requests) seems doable and would make the message a lot more useful than a generic "rate limited" line. Agreed on not logging full request/headers. Given both of these, it sounds like a good first PR would be: a bounded, deduplicated warning at the generator retry boundary, scoped to a single process (skip the multiprocessing/backoff-metrics aggregation for now per jmartin-tech's note), that reports the exhausted limit type where available and cumulative wait time separate from generation time. Happy to take a crack at this if it's a welcome contribution, scoped to just the warning, no behavior change. Let me know if that scope sounds right or if I'm missing something. |
|
One thing that might also help is showing the 429 count and total time spent waiting in the final output. That would make it much easier to tell whether a long run is actually slow inference or mostly provider throttling.
|
|
Thanks, this is helpful — agree on all three. The warning will honor Retry-After (plus remaining/reset headers where the provider sends them) instead of relying only on the backoff default, track backoff wait separately from run time so the estimate stays honest, and include the 429 count + total throttled time in the final report. |
Uh oh!
There was an error while loading. Please reload this page.
Hey — ran into something while testing garak against a Groq-hosted model and wanted to flag it, since it wasn't obvious what was actually happening until I dug into the logs.
Setup: GitHub Codespaces (CPU-only), garak v0.17.0, Groq free-tier key.
garak --model_type groq --model_name openai/gpt-oss-20b --probes promptinject.HijackHateHumans --parallel_attempts 4
The progress bar showed an estimated 20+ hours remaining for a probe that only needed 5 actual model completions. At first I assumed it was a slow local detector model, but checking garak.log showed the real cause: repeated HTTP 429s from Groq, each triggering GeneratorBackoffTrigger. Pulling the rate-limit headers directly confirmed why — my account's limit is 8,000 tokens/minute, and --parallel_attempts 4 was enough to blow through that almost immediately, so it fell into a retry/backoff loop that compounded into a huge runtime estimate.
Not sure if this is worth a formal bug label since it's really a provider-tier limitation interacting with garak's concurrency defaults rather than a garak bug per se — but from a user's side, it just looks like "garak is extremely slow," with nothing pointing at rate-limiting as the actual cause. Saw #2189 is in similar territory (parallel_attempts causing bad behavior that's hard to diagnose from the CLI output alone), though that one's a resource/hang issue rather than a rate-limit one.
Would something like a CLI warning when repeated 429s are detected (e.g. "rate limited by provider — consider lowering --parallel_attempts") be in scope? Happy to share the full log if useful, or take a crack at a small PR if that's a welcome contribution.
All reactions