-
Notifications
You must be signed in to change notification settings - Fork 136
Heritage
Sparkler began at USC IRDS as a Nutch-like crawler on Apache Spark with a Solr CrawlDB. This rewrite is still USCDataScience/sparkler. It is not the Spark cluster product, and it is not a continuing IRDS lab stack.
Old UI/control trees: sparkler-ui and sce-domain-discovery are archived. Git history on main (pre-rewrite) still has the Scala engine.
| 2016–2023 | Now |
|---|---|
| inject → generate → fetch → parse → upsert | Same loop, one process |
| Solr CrawlDB | Solr 10 CrawlDB (not ES) |
| Tika parse | Tika-Python + HTML outlinks |
| Host grouping / FairFetcher | Host-fair generate, delay on apex host |
| regex + same-host plugins |
regex-urlfilter.txt + --same-host (www = apex) |
| robots.txt (Nutch port, unfinished) | urllib robotparser
|
| Domain discovery (SCE labels) | Documents labels + keyword scorer |
| Banana / Solr iframe dashboard | Vue Documents + D3 Stats (heatmap, donuts, bubbles, treemap) |
-i -1 until empty (broken) |
-i -1 / until done; same-host leftovers marked FILTERED |
Not coming back in v1: Apache Spark, Elasticsearch, Docker/dockler.sh, Kafka dump, Databricks, pf4j Chrome/HtmlUnit/jBrowser plugins, k8s/helm/juju/Gitpod, ArangoDB, DuckDuckGo scrape, Flask-restplus, React 16 CRA, Banana, SolrCloud, com.kythera.
Spark remains a possible later backend for a 100M-URL crawl. The product is a workstation you run with bin/sparkler.
Historic wiki pages: Old.
Inspired by USC IRDS Sparkler.
Old (2016–2023)