Skip to content

v-code01/retrystorm

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

retrystorm: backoff without jitter is a self-inflicted DDoS

When a service falls over, its clients retry. The standard advice is exponential backoff: wait 1 second, then 2, then 4, and so on, so you stop hammering a service that is down. What the backoff schedule alone does not fix is that every client that failed at the same moment also retries at the same moments. Ten thousand clients all wait one second and then all retry in the same instant, and that synchronized wave knocks the recovering service back over. This measures the wave, and shows that the fix is not a better backoff, it is jitter.

Measured (peak retries in any 20 ms window, N clients failing together, BCa 95% CI, 11 seeds)

N no jitter full jitter equal jitter decorrelated
1,000 1,000 39 [37, 42] 51 [48, 51] 20 [18, 20]
10,000 10,000 327 [314, 327] 438 [431, 439] 150 [142, 152]
100,000 100,000 3,148 [3,113, 3,155] 4,111 [4,100, 4,114] 1,347 [1,327, 1,357]

The no-jitter column is the whole fleet: peak equals N exactly, at every scale, because the clients retry in lockstep and the entire population lands in one window. That is the self-inflicted denial of service, and no amount of backoff-schedule tuning changes it, because every client follows the same schedule. Adding jitter breaks the lockstep. Full jitter (retry at a uniform random point in the backoff interval) cuts the peak about thirty times. Decorrelated jitter (each sleep drawn between the base and three times the last sleep) cuts it about sixty-seven times and is the best of the four. Equal jitter is the weakest, because it still pins half of every interval to a fixed offset.

The ordering holds at every scale: decorrelated beats full beats equal beats none, and the gap between none and any jitter is an order of magnitude or two.

The takeaway

Exponential backoff is not what protects a recovering service from a retry storm, jitter is. Backoff decides how often one client retries; jitter decides whether ten thousand clients retry together. Without it, your own clients re-DDoS you the instant you come back up, and it looks exactly like an attack. Add full jitter at least, and decorrelated jitter if you can, so the fleet that failed together does not retry together.

Reproduce

./run.sh

Needs a recent Rust (edition 2021) and Python 3 (standard library only). Counts, no timing; the numbers are stable. Peak scales with the window resolution (a finer window makes the no-jitter spike relatively worse), which is stated in scope.

Files

  • src/lib.rs: the retry-schedule simulation and the four jitter strategies, with the tests (no jitter spikes the fleet, jitter cuts the peak, decorrelated wins).
  • src/bin/bench.rs: peak retries for a given (N, strategy, seed).
  • sweep.py: the N-by-strategy sweep with BCa CIs.
  • PREREG.md: the pre-registration.
  • run.sh: reproduces everything.

Scope

Simulation with exact retry counts. The claim is the lockstep spike without jitter, the order-of-magnitude peak reduction with jitter, and the strategy ordering.

License

MIT.

About

Exponential backoff without jitter synchronizes a failed fleet into one retry spike (peak equals N, a self-inflicted DDoS); full jitter cuts the peak about 30x and decorrelated jitter about 67x. Jitter, not the backoff schedule, flattens the storm.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors