Skip to content

http: add 429 rate-limit handling with host-level cooldowns#3145

Merged
liquidsec merged 4 commits into
devfrom
http-429-handling
Jun 17, 2026
Merged

http: add 429 rate-limit handling with host-level cooldowns#3145
liquidsec merged 4 commits into
devfrom
http-429-handling

Conversation

@liquidsec

Copy link
Copy Markdown
Collaborator

Summary

  • When the http module's batch stream receives a 429, it defers the event and sets a per-host cooldown using the Retry-After header (default 30s, capped at 60s)
  • Deferred events re-enter the queue after their cooldown expires, bypassing dedup via is_incoming_duplicate override
  • Gives up after 3 retries per URL to prevent infinite loops
  • Overrides finished property so deferred events keep the scan alive during cooldowns
  • Uses existing 429_sleep_interval / 429_max_sleep_interval config values from the API 429 handler
  • Non-blocking: other hosts in the same batch proceed normally while one host is cooled down

Tested against a local rate-limiting server (429 after 5 requests, 3s Retry-After) and against stackoverflow.com under Cloudflare rate limiting.

@github-actions

github-actions Bot commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

🚀 Performance Benchmark Report

⚠️ No current benchmark data available

This might be because:

  • Benchmarks failed to run
  • No benchmark tests found
  • Dependencies missing

@codecov

codecov Bot commented Jun 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.47368% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (b795c7c) to head (36013dd).
⚠️ Report is 79 commits behind head on dev.

Files with missing lines Patch % Lines
bbot/modules/http.py 87% 14 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             dev   #3145     +/-   ##
=======================================
- Coverage     90%     90%     -0%     
=======================================
  Files        449     453      +4     
  Lines      44435   46024   +1589     
=======================================
+ Hits       39762   41152   +1390     
- Misses      4673    4872    +199     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ausmaster ausmaster self-requested a review June 9, 2026 02:46
When a batch response returns 429, defer the event and set a per-host
cooldown using the Retry-After header (or 30s default). Re-queue after
cooldown expires via is_incoming_duplicate bypass. Give up after 3
retries per URL.
… drop exhausted 429s

- Exhausted-retry 429s now skip processing instead of emitting as scan results
- _defer_event deduplicates to prevent double-deferral from paired probes
- _wakeup_pending flag prevents accumulation of redundant wakeup tasks
- Retry count keyed by event hash instead of URL (OPEN_TCP_PORT gets 3 retries total)
- Retry count entries cleaned up after max retries
- Add TestHTTP_429_retry and TestHTTP_429_max_retries
@liquidsec liquidsec force-pushed the http-429-handling branch from b0ea387 to dfc98f1 Compare June 14, 2026 17:15
@liquidsec liquidsec mentioned this pull request Jun 15, 2026
28 tasks
- Fix wakeup-latency ordering: earlier cooldowns now preempt pending wakeups
- Prune expired cooldowns and retry counts to avoid unbounded growth
- Fix 429 test to actually exercise the module's defer/cooldown/retry path
@liquidsec liquidsec merged commit 9cc4809 into dev Jun 17, 2026
15 checks passed
@liquidsec liquidsec deleted the http-429-handling branch June 17, 2026 00:01
@liquidsec liquidsec mentioned this pull request Jun 17, 2026
@ausmaster ausmaster added this to the BBOT 3.0 - blazed_elijah milestone Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants