Skip to content

fix(llm, cli, config): Jitter retry backoff so sessions don't collide - #1080

Merged
JeanMertz merged 5 commits into
mainfrom
jitter-retry
Sep 4, 2026
Merged

fix(llm, cli, config): Jitter retry backoff so sessions don't collide#1080
JeanMertz merged 5 commits into
mainfrom
jitter-retry

Conversation

@JeanMertz

Copy link
Copy Markdown
Collaborator

Retry waits now carry a random extra of up to a quarter of the delay, so several JP sessions sharing one API key stop resuming in the same instant after being rate limited together. Cerebras enforces its limits per organization and answers a 429 with retry-after: 60, so every session that hit the limit slept exactly 60000ms, woke together, and walked into the limit again — burning the whole retry budget without any of them getting through.

The base_backoff_ms documentation has described this jitter since the setting was introduced, down to a stated 0-500ms range, but nothing ever implemented it. That text reaches users through the generated config.toml and the exported JSON schema, so it was a promise the program did not keep. It now describes what the code does.

Jitter is added after the backoff ceiling rather than folded inside it. Folding it in means it disappears exactly when it matters, since every caller pinned at the ceiling gets the same number. It is also only ever added, never subtracted: a retry_after is the minimum the provider asked for, and waiting less walks straight back into the limit. One consequence worth knowing about is that an individual wait can now exceed max_backoff_secs by up to a quarter.

Both retry layers pick their delay through the new jp_llm::retry_delay, which replaces the identical retry_after-or-exponential match they each carried. exponential_backoff stays pure and keeps its exact tests.

Retry waits now carry a random extra of up to a quarter of the delay, so
several JP sessions sharing one API key stop resuming in the same instant
after being rate limited together. Cerebras enforces its limits per
organization and answers a 429 with `retry-after: 60`, so every session
that hit the limit slept exactly 60000ms, woke together, and walked into
the limit again — burning the whole retry budget without any of them
getting through.

The `base_backoff_ms` documentation has described this jitter since the
setting was introduced, down to a stated 0-500ms range, but nothing ever
implemented it. That text reaches users through the generated
`config.toml` and the exported JSON schema, so it was a promise the
program did not keep. It now describes what the code does.

Jitter is added after the backoff ceiling rather than folded inside it.
Folding it in means it disappears exactly when it matters, since every
caller pinned at the ceiling gets the same number. It is also only ever
added, never subtracted: a `retry_after` is the minimum the provider
asked for, and waiting less walks straight back into the limit. One
consequence worth knowing about is that an individual wait can now exceed
`max_backoff_secs` by up to a quarter.

Both retry layers pick their delay through the new `jp_llm::retry_delay`,
which replaces the identical `retry_after`-or-exponential match they each
carried. `exponential_backoff` stays pure and keeps its exact tests.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
@JeanMertz
JeanMertz merged commit f53c762 into main Sep 4, 2026
21 checks passed
@JeanMertz
JeanMertz deleted the jitter-retry branch September 4, 2026 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant