What's new in v1.5.9
Bug fix: Missing H1 / Title / Meta counts no longer inflated by broken pages
The Issues tab was counting 4xx and 5xx pages as "Missing H1", "Missing Title", and "Missing Meta Description" — because the crawler doesn't read body content for error responses, so those fields are always blank. On WordPress sites with any broken internal links, this could show dozens of phantom issues even when every live page was fine. Fixed: all content-quality issue counts now apply to HTTP 200 pages only, matching how OG tag issues already worked.
MCP: fetch_batch — bulk page fetching
Fetches a list of URLs concurrently and returns each page's content as clean Markdown in a single call. Useful for bulk content extraction, competitor research, or comparing many pages at once without running a full crawl.
MCP: crawl_site curl mode — Cloudflare bypass
crawl_site now accepts useCurl: true to use the system curl binary instead of Node.js for HTTP fetching. This bypasses Cloudflare's TLS fingerprinting that blocks undici/Node.js requests and returns 403s or 429 CAPTCHA challenges. Requires curl on PATH (standard on macOS and Linux).
MCP: Live progress notifications during crawl_site
Claude Code now receives real-time progress updates while crawl_site is running — you'll see pages crawled and queue size update in the tool call rather than waiting in silence until the crawl finishes.
MCP: DB corruption recovery
If the Crawly database file is corrupted (e.g. after a crash mid-write), the MCP server now renames the broken file instead of crashing, then starts fresh.
Engine: Rate-limit improvements
Retry-Afterheader values from 429 responses are now capped at 30 seconds- Per-domain backoff sleep now includes a small random jitter to reduce thundering-herd effects