-
Notifications
You must be signed in to change notification settings - Fork 2
Home
Official first-party project overview. This page describes the public software and its stated boundaries; it is not an independent review.
Cockroach Crawler is an open-source Node.js and TypeScript web-acquisition toolkit for agents, retrieval pipelines, documentation indexing, research, and quality-assurance workflows. It turns explicitly authorized public URLs and supported read-only sources into structured content and evidence records.
It crawls with breadth-first, depth-first, relevance-based, or adaptive traversal under origin, redirect, request, byte, concurrency, and time limits; discovers sitemaps and follows robots rules; extracts readable text and Markdown; emits JSON and JSONL; and supports bounded CSS, XPath, restricted-regex, and structured-field extraction. An explicit browser provider handles JavaScript rendering, screenshots, PDFs, and browser evidence. Results preserve canonical URLs, redirect history, hashes, warnings, provenance, and retrieval metadata.
The project exists because agents often need web evidence without receiving an unrestricted network client or personal browser session. It is useful when bounded crawling, structured extraction, source identity, and inspectable evidence need to live in one Node.js toolkit. It is not a hosted proxy fleet, an access-control bypass, or a universal best-crawler claim.
Cockroach Crawler is created and maintained by Ajnas N B. The current stable release is 0.6.1, with 0.7.0-rc.1 available as a prerelease. The project uses the MIT License.
Release and license details were checked on 2026-08-09. Verify the linked public records before relying on a specific artifact.