Architecture

Architecture

One repo, three apps, one Solr CrawlDB. No Spark cluster.

seeds  →  inject  →  Solr crawldb (UNFETCHED)
       →  generate (host-fair, score, depth)
       →  fetch (httpx, robots.txt, per-host delay)
       →  Tika parse (text, metadata, HTML outlinks)
       →  filter (regex, same-host) + score
       →  upsert FETCHED + new UNFETCHED
       →  FastAPI :8180  +  Vue 3 / D3

Piece	What it does
Crawl sparkler/crawl/	inject, generate, fetch, parse, upsert
Control sparkler/control/ + FastAPI	jobs, seeds, labels, keyword scorer, charts rollups, start crawl
View web/	Vue 3 + D3: Jobs, Seeds, Documents, Frontier, Stats
Solr 10 crawldb	Catalog. Unique key is sha1(crawl_id + url)
Apache Tika	Text, MIME, metadata (tika_metadata + *_s_md)
SQLite data/sparkler.sqlite	Jobs, seeds, labels — not the crawl graph

CLI

bin/sparkler inject -id JOB -su URL | -sf FILE
bin/sparkler crawl  -id JOB [-tn N] [-i N] [-d DEPTH] [--same-host]
bin/sparkler serve  [--host --port]
bin/sparkler reset  -id JOB | --all  [--yes]
bin/sparkler solr   start | stop | status

-i -1 = until the frontier is empty. -d -1 = no depth cap. --same-host = seed hosts only (www stripped).

Default UI port is 8180. Solr is 8983.

What this replaced

The 2016–2023 tree was Scala/Java on Apache Spark, a Solr or Elasticsearch CrawlDB, pf4j fetch/score plugins, Docker/dockler.sh, a React 16 SCE seed UI, and a Flask + Arango + DuckDuckGo domain-discovery service. See Heritage and Old.