Crawling

Crawling

A crawl walks a whole site and collects its content, sharded across your agent fleet. Where a workflow is "do this exact task", a crawl is "go and read all of this".

Starting one

Harvest → New. Give it a starting URL and describe what you want. The coordinator derives a scope — which paths to follow, how deep, what counts as content — and shows you a preview of that scope before anything runs.

Options worth knowing:

Option	Values	What it does
render_mode	auto | http | browser	http is a plain fetch and is much cheaper. browser runs a real browser for pages that need JavaScript. auto decides per page.
ocr_mode	auto | off | force	OCR policy for PDFs, office files and images. See Documents and OCR.
max_depth	integer	Link-discovery depth. Unset lets the derived depth win.
include_paths / exclude_paths	patterns	Narrow or carve out sections of the site.

Content extraction is configurable too — which selectors to keep or drop, whether to include comments, and what to treat as the main body.

While it runs

The crawl detail view shows live progress: pages fetched, queued and failed, and a feed of what has just come in. Work is distributed across every agent with capacity, so adding agents makes a crawl finish sooner.

Non-HTML content

PDFs, Word/Excel/PowerPoint files, images and scanned pages are read by the document extractor, which ships with the coordinator. If DOC_EXTRACT_URL is unreachable from your agents, that content is skipped silently — the crawl still reports success. If you expected PDFs and got none, check that first.

Results

Everything lands in a dataset: searchable, queryable across runs, and exportable.

Being a good citizen

• Targets resolving to private or loopback addresses are refused unless ALLOW_PRIVATE_TARGETS=true.
• Per-host dispatch is capped by TARGET_DOMAIN_RATE_LIMIT / TARGET_DOMAIN_RATE_WINDOW_SECS, so a wide crawl cannot hammer one origin.
• Crawls browse from your IPs. That is a feature — no shared proxy pool poisoning your reputation — but it also means the target sees you.