Skip to content

feat(retry): introduce retry budget in retry layer#7430

Open
dentiny wants to merge 3 commits intoapache:mainfrom
dentiny:hjiang/introduce-retry-budget
Open

feat(retry): introduce retry budget in retry layer#7430
dentiny wants to merge 3 commits intoapache:mainfrom
dentiny:hjiang/introduce-retry-budget

Conversation

@dentiny
Copy link
Copy Markdown
Contributor

@dentiny dentiny commented Apr 27, 2026

Which issue does this PR close?

Closes #7376

Rationale for this change

Current retry layer's implementation supports min/max retry duration, max count and jitter; when used together with timeout layer, we could also cap the upper bound for operation duration.
But in production, we've also observed retry storm, which degrades already overloaded / bad service.

Some implementation reference in the industry:

What changes are included in this PR?

This PR adds a no-op retry budget to retry layer:

  • It's kept no-op by default to keep the current retry behavior
  • It's a trait, so users could implement their own budget
    • Both deposit and withdraw take operation, so the trait allows either shared budget across all operations, or per-operation; it could also be used for metrics purpose
    • Could be a followup item to provide a default budget implementation so users could leverage out-of-the-box

Are there any user-facing changes?

Yes

AI Usage Statement

Opus 4.7 made most of the code change, I did the code review

@dentiny dentiny requested a review from Xuanwo as a code owner April 27, 2026 08:16
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. releases-note/feat The PR implements a new feature or has a title that begins with "feat" labels Apr 27, 2026
@dentiny
Copy link
Copy Markdown
Contributor Author

dentiny commented Apr 27, 2026

make_bucket: test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   141  100   141    0     0    934      0 --:--:-- --:--:-- --:--:--   940
./mc: line 1: a: No such file or directory

CI failure doesn't seem to be related

@dentiny
Copy link
Copy Markdown
Contributor Author

dentiny commented Apr 28, 2026

---- behavior::test_list_empty_dir ----
test panicked: assertion `left == right` failed: only return the dir itself
  left: 0
 right: 1

hmm not seem to relate to my changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

releases-note/feat The PR implements a new feature or has a title that begins with "feat" size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

new feature: introduce retry budget to prevent retry storm

1 participant