Home

Sparkler

A crawl workstation: View, Control, and Crawl.

Inject seeds. Fetch politely. Parse with Tika. Store the frontier and pages in a Solr CrawlDB. Label pages relevant / not / highly so the next generate pass prefers the right hosts.

git clone https://github.com/USCDataScience/sparkler.git
cd sparkler
bin/sparkler inject -id demo -sf demo/seeds.txt
bin/sparkler crawl  -id demo --same-host -i -1
bin/sparkler serve          # http://127.0.0.1:8180/

No Docker. No Spark. No Elasticsearch. No ArangoDB. No Banana.

Start at Getting started. How this relates to the 2016–2023 Spark crawler: Heritage. Historic wiki pages live under Old.