v2.24.1

Security fixes. Upgrading is recommended for anyone who crawls sites they don't control or runs the MCP server.

Behavior changes

• HTML pages are weighed before parsing (tags, attributes and size). A page heavier than about 1.5 million units is skipped as "too complex" and counts as a failed page. The largest real docs page we know of, the WHATWG HTML spec, weighs about 1 million. All page parsing in a process (crawl workers, MCP crawl jobs, get_page, add and draft_site) shares one budget, so very large pages wait for each other instead of running at once.
• get_page now honors max_page_size_bytes instead of a fixed 50 MiB.
• Sitemaps read by add and draft_site are capped at 50 MiB after decompression, and URLs longer than 2,048 bytes are skipped.
• Crawls follow sitemap indexes at most two levels deep and fetch at most 1,000 sitemaps per site per run.
• Sitemaps, including Sitemap: lines in robots.txt, are only fetched from the site's allowed_domain (or, for add and draft_site, the start page's host), and a sitemap that redirects to another host is dropped.
• The SSRF guard now allows only globally reachable addresses. Documentation, benchmarking and reserved ranges, and 168.63.129.16, are blocked along with the private, loopback, link-local, CGNAT and multicast ranges that were blocked before.

Security

• golang.org/x/net 0.62.0, for GO-2026-6860 (CVE-2026-78661) and GO-2026-6866 (CVE-2026-97045). The first made html.Parse take quadratic time on HTML with many adjacent text nodes, and every page doc-scraper parses goes through it, so a crafted page could tie up a crawl worker or get_page. The second affects html.Render on raw-text elements, which readability extraction uses.

Also in this release

• A page that goes through readability extraction is no longer copied first when it has no breadcrumb markers, so it is held in memory once instead of twice.