Crawl

Crawl

Polite, single-process, Nutch-shaped loop. Not a Spark job.

1. inject — seeds become UNFETCHED at depth 0.
2. generate — take up to topn UNFETCHED URLs, fair across hosts, sort page_score desc, discover_depth asc. With same host, only seed hosts (www and apex). With a max depth, only discover_depth <= depth.
3. fetch — httpx, User-Agent: Sparkler/1.0, robots.txt, 1s delay per apex host.
4. parse — Tika for text and metadata; HTML href for outlinks. Junk tel: / phone-as-port hrefs are skipped.
5. filter — conf/regex-urlfilter.txt (skip images, scripts, mailto, PDFs by default). Same-host if requested.
6. upsert — FETCHED page + new UNFETCHED outlinks at depth+1.
7. same-host finish — leftover UNFETCHED URLs that are not on a seed host are marked FILTERED so the frontier can go to 0.

One iteration is one generate+fetch batch (UI default 50 URLs). That is why a first click can show 1 fetched and dozens waiting. until done / -i -1 keeps going until generate returns nothing.

Whole site

bin/sparkler inject -id mai -su 'https://mattmann.ai/'
bin/sparkler crawl  -id mai --same-host -i -1

www.mattmann.ai counts as mattmann.ai. http:// and https://www. of the same path are still two catalog rows (canonical collapse is not in v1).

Blog ?e-page= pagination increments discover_depth each hop. The Stats heatmap bins deep pages as 16+; the Solr field still has the raw depth.

Extra sites

Add those URLs as seeds, keep same host on, until done.

Or uncheck same host and set depth to 2 or 3. Depth unlimited with same host off will wander.

Config

conf/sparkler.yaml — Solr host/port/core, serve bind, delay, timeout, user-agent, robots, default same-host.

conf/regex-urlfilter.txt — Nutch-style + / - rules.