Skip to content

Exponential backoff overflows after 64 failures, collapsing to a 1-minute retry storm #369

Description

@forkwright

Finding

current_interval_minutes computes the backoff as base_interval_minutes * 2u64.pow(self.failure_count). failure_count grows on every consecutive failure with saturating_add(1) and is never bounded before being used as the exponent. Once it reaches 64, 2u64.pow(64) overflows u64: it panics in debug builds and wraps to 0 in release builds. On wrap the backed-off interval becomes base * 0 = 0, .min(max_backoff_minutes) keeps it at 0, and the downstream jitter floor (.max(1)) clamps the result up to a 1-minute interval. The intended ever-lengthening backoff therefore inverts into a permanent tight retry loop against a failing remote.

Evidence

crates/komide/src/scheduler.rs:39 (the overflowing exponent):

let backed_off = self.base_interval_minutes * 2u64.pow(self.failure_count);

crates/komide/src/scheduler.rs:182 (failure_count increments with no ceiling):

state.failure_count = state.failure_count.saturating_add(1);

saturating_add only protects the counter's own integer range; it places no bound on the exponent, so failure_count freely reaches 64 and beyond. The jitter floor at crates/komide/src/scheduler.rs:64 (.max(1) as u64) is what turns the wrapped 0 into a 1-minute interval rather than 0.

Why this matters

A feed whose host is persistently unreachable accumulates failures indefinitely. After 64 of them the scheduler polls that dead host once per minute forever instead of backing off, producing a steady, predictable beacon of outbound requests and log entries. Under a counter-surveillance threat model that regular, machine-generated traffic pattern is exactly the kind of fingerprint an on-path adversary can use to identify the node and its subscriptions, and it defeats the traffic-minimization the backoff was meant to provide. In debug or test builds the pow overflow panics outright, taking down every feed managed by that scheduler instance.

Desired correction

Clamp the exponent before the shift, e.g. let shift = self.failure_count.min(63); let backed_off = self.base_interval_minutes.saturating_mul(2u64.pow(shift));, using saturating multiplication so a large base interval cannot overflow either. The interval must monotonically rise to max_backoff_minutes and stay there. Done when: a test driving failure_count to 100 returns max_backoff_minutes with no panic and never returns 0, in both debug and release builds.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions