This release adds per-request control (callbacks, JSON bodies, custom headers), makes retry/delay/dedup/cookies fully configurable, redesigns the stats API, and hardens security across redirects, storage, exporters, and offsite filtering.
Breaking change
- StatsCollector API redesigned to typed verbs. inc_value / set_counter / set_meta / get_value / get_stats are replaced by inc(key, count=1), set, max, min, label, get, and snapshot. set/max/min raise TypeError on non-numeric values; label requires a string. Update custom stats handlers accordingly.
New features
- Per-request callbacks — Request(callback=..., cb_kwargs=...) (and follow(callback=, cb_kwargs=)) route a response to any spider method instead of always parse. Callbacks are stored by name so requests stay serializable.
- json= request bodies — Request(json={...}) serializes the body and sets Content-Type: application/json automatically (mutually exclusive with body=). No more manual orjson.dumps(...).
- follow(headers=...) — pass per-request headers when following links.
- Configurable retries — RETRY_ENABLED, RETRY_HTTP_CODES, RETRY_PRIORITY_ADJUST, RETRY_BACKOFF_BASE / RETRY_BACKOFF_MAX / RETRY_BACKOFF_JITTER, plus per-request dont_retry and max_retry_times meta.
- RANDOMIZE_DELAY — jitter the per-domain download delay to look less robotic.
- COOKIES_ENABLED — toggle cookie handling.
- Query-aware deduplication — IGNORE_QUERY_PARAMS / KEEP_QUERY_PARAMS control which query parameters count toward a request fingerprint (mutually exclusive).
- New signals — request_failed (final give-up only), spider_idle (fires until the scheduler is truly drained), headers_received (pre-body), and item_error.
Security
- Strip Authorization and Cookie headers on cross-host redirects.
- Reject path traversal outside the configured storage root.
- Neutralize CSV formula injection in exporters.
- Offsite filter now rejects parent/sibling domains when only a subdomain is allowed.
Fixes & hardening
- Fixed "dead settings" — MAX_DEPTH and other settings were declared but never wired into their middleware via from_crawler; they now take effect.
- Centralized validation in Settings.post_init (MAX_DEPTH, DELAY_PER_DOMAIN, RETRY_*, query-param exclusivity).
- Camoufox: fixed persistent_context launch by reusing the single context; the CAMOUFOX_ABORT_REQUEST predicate is now actually applied (awaitable predicates supported).
- Engine: a pipeline error drops a single item, not the whole response.
- Pipelines: re-raise unexpected errors instead of silently dropping items.
- Validation: keep falsy values (0, "", False); drop only None/missing required fields.
- Scheduler: emit request_dropped when the queue is full.
Internals
- Engine owns the downloader middleware chain, wires spider-exception handling, and runs pipelines on the data path.
- Crawler owns pipeline construction; canonical PIPELINES loading.