-
Notifications
You must be signed in to change notification settings - Fork 4
resilient system
The Resilience System is AtDork's intelligent fault‑tolerance engine.
It ensures that large‑scale OSINT operations continue smoothly even when search engines block you, proxies fail, or network connections drop.
Activated with --resilient, it coordinates seven specialised modules under core/case/ to handle every failure scenario automatically.
| Scenario | Without Resilience | With Resilience |
|---|---|---|
| Google returns HTTP 429 (Rate Limit) | All subsequent queries fail | Switches to Startpage, cools down Google for 120s |
| Proxy dies mid‑batch | Batch crashes | Rotates to next healthy proxy |
| All backends blocked | Scan fails completely | Falls back through 5+ search engines until one works |
| Temporary network timeout | Query fails permanently | Retries up to 3x with exponential backoff |
| Backend returns empty results | Wastes retries on dead backend | Immediately switches to next backend |
Classifies every exception into one of five categories:
| Category | Triggered By | Resilience Action |
|---|---|---|
TRANSIENT |
Timeout, connection reset | Retry with backoff |
RATE_LIMIT |
HTTP 429, "too many requests" | Long pause, switch backend |
BLOCKED |
HTTP 403, CAPTCHA | Switch backend + rotate proxy |
PROXY_FAIL |
Proxy timeout, cooldown, auth error | Rotate to next proxy |
FATAL |
Code bug, parsing error | Abort immediately |
After a backend/proxy fails 3 times consecutively, the circuit breaker opens and refuses further requests for 120 seconds.
This prevents AtDork from wasting time on resources that are clearly unavailable.
Backend Google fails 3 times → Circuit OPEN for 120s
After 120s → Half‑open: one request allowed
If that succeeds → Circuit CLOSED (healthy again)
If it fails → Circuit OPEN for another 120s
Decides the best action based on the error category and current state of backends/proxies.
| Error Category | Decision |
|---|---|
RATE_LIMIT |
Switch to next healthy backend |
BLOCKED |
Switch backend + rotate proxy |
PROXY_FAIL |
Rotate proxy; if all proxies dead, cooldown |
TRANSIENT |
Retry same resource |
FATAL |
Abort with clear error |
Retries failed requests with increasing delays to avoid overwhelming servers.
| Attempt | Delay (approx) |
|---|---|
| 1st | 2s |
| 2nd | 4s |
| 3rd | 8s |
Jitter (random variance) prevents all threads from retrying at the exact same moment.
Adjusts delay per backend based on real‑time response codes.
- Success (200 + results): Delay decreases by 10% (down to 0.1s minimum).
- Rate limit (429): Delay doubles (up to 60s maximum).
- Other errors: Delay unchanged.
At the end of a batch, recommendations are displayed:
Rate Limiter Recommendations:
google: 8/20 rate‑limited. Increase delay to 5.0s or switch backend.
startpage: 0/15 rate‑limited. Everything healthy at 0.3s.
When --strict is active, IP Guard periodically checks that your real IP is not being exposed.
- Establishes a baseline (IP visible through proxy/Tor).
- Checks every few queries that the visible IP hasn't changed to your real IP.
- Inspects response headers (
X‑Forwarded‑For,CF‑Connecting‑IP) for leaks. - If a leak is detected, AtDork halts immediately with a detailed panic message.
Maps error categories to recommended recovery actions for consistent behaviour across all modules.
# Basic resilience
atdork --batch-file dorks.txt --resilient
# Resilience + adaptive rate limiting
atdork --batch-file dorks.txt --resilient --adaptive-delay
# Full protection (recommended for sensitive operations)
atdork --batch-file dorks.txt --resilient --adaptive-delay --ip-guard --strict --proxy-file proxies.txt| Scenario | Recommended Flags |
|---|---|
| Small test (< 5 queries) | None needed |
| Medium batch (10–50 queries) | --adaptive-delay |
| Large batch (> 50 queries) | --resilient --adaptive-delay |
| With free proxies |
--resilient --adaptive-delay (mandatory) |
| Anonymity critical | --resilient --ip-guard --strict --tor |
| Unattended overnight | --resilient --adaptive-delay |