v3.7.0: Robustness & SPN2 API Alignment
Release Notes
Robustness fixes and better alignment with the SPN2 API's documented limits. Fixes correctness bugs where transient failures could permanently lose in-flight jobs, and adds defensive measures against hung connections and API overload.
No breaking changes.
What's New
- HTTP timeouts on all requests (10s connect, 60s read) — hung connections no longer block indefinitely.
- Poll failure resilience — a single failed poll no longer kills all pending jobs. Retries with backoff up to 5 consecutive failures.
- Concurrency cap at 10 in-flight jobs, under the API's ~12 session limit. Prevents user-session-limit errors from burning retry attempts.
- Sitemap index recursion — files are now followed automatically (depth limit 5).
- XML hardening via defusedxml.
- Fixed backoff factor — was accidentally using the rate limit (15s) as the retry backoff, producing 15s/30s/60s sleeps. Now 1s/2s/4s/8s/16s.
- Fixed exception handling — no-job-id responses (server load) no longer count against a URL's retry budget.
- Job timeout reduced from 2h to 1h to match the API's status availability window.
- 429 added to retry list — urllib3 honors Retry-After automatically.
- python -m wayback_machine_archiver support.
- Local sitemaps now require the file:// prefix as documented (bare /path no longer accepted).
- Batch status chunking at 50 job IDs per request.