http: add 429 rate-limit handling with host-level cooldowns#3145
Merged
Conversation
Contributor
🚀 Performance Benchmark Report
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev #3145 +/- ##
=======================================
- Coverage 90% 90% -0%
=======================================
Files 449 453 +4
Lines 44435 46024 +1589
=======================================
+ Hits 39762 41152 +1390
- Misses 4673 4872 +199 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
When a batch response returns 429, defer the event and set a per-host cooldown using the Retry-After header (or 30s default). Re-queue after cooldown expires via is_incoming_duplicate bypass. Give up after 3 retries per URL.
… drop exhausted 429s - Exhausted-retry 429s now skip processing instead of emitting as scan results - _defer_event deduplicates to prevent double-deferral from paired probes - _wakeup_pending flag prevents accumulation of redundant wakeup tasks - Retry count keyed by event hash instead of URL (OPEN_TCP_PORT gets 3 retries total) - Retry count entries cleaned up after max retries - Add TestHTTP_429_retry and TestHTTP_429_max_retries
b0ea387 to
dfc98f1
Compare
- Fix wakeup-latency ordering: earlier cooldowns now preempt pending wakeups - Prune expired cooldowns and retry counts to avoid unbounded growth - Fix 429 test to actually exercise the module's defer/cooldown/retry path
ausmaster
approved these changes
Jun 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
is_incoming_duplicateoverridefinishedproperty so deferred events keep the scan alive during cooldowns429_sleep_interval/429_max_sleep_intervalconfig values from the API 429 handlerTested against a local rate-limiting server (429 after 5 requests, 3s Retry-After) and against stackoverflow.com under Cloudflare rate limiting.