Getting Started

Getting started

Needs Java 21+ (Solr 10), Python 3.10+, and Node 18+ to build the UI. Tika can use Java 11+; Sparkler points Solr at Java 21 when it finds Homebrew openjdk@21.

git clone https://github.com/USCDataScience/sparkler.git
cd sparkler
python3.12 -m venv .venv
.venv/bin/pip install -e .
cd web && npm install && npm run build && cd ..

bin/sparkler inject -id demo -sf demo/seeds.txt
bin/sparkler crawl  -id demo --same-host -i 1
bin/sparkler serve

Open http://127.0.0.1:8180/

bin/sparkler uses .venv when it exists.

Solr 10 slim is downloaded on first use into data/ (gitignored) and bound to 127.0.0.1:8983. The app is 127.0.0.1:8180. Those ports are chosen so they do not collide with DRAT (9000), ImageCat live (9100), BigTranslate (9200 / Solr 8985), or Meridian (8090).

bin/sparkler inject -id news -su 'https://example.com/'
bin/sparkler crawl  -id news --same-host -i -1
bin/sparkler reset  -id news --yes
bin/sparkler solr   status

-i -1 means keep fetching until the frontier is empty (capped at 500 iterations). --same-host stays on the seed site; www.example.com and example.com count as one site.

What you should see

Jobs, Seeds, Documents, Frontier, Stats.

• Documents — title, URL, status, depth, Tika Content-Type, Tika metadata table (click a row), extracted text, relevance labels.
• Frontier — UNFETCHED URLs, highest score first.
• Stats — status and host charts.

same host (on) = crawl the seed site. until done = do not stop after one batch. depth = link hops from a seed (1 is the homepage plus its links). Add extra seed URLs (a ucla.edu page, for example) to include those hosts while same host is on. Uncheck same host to follow off-site links, limited by depth.

The 2016–2023 Spark / Docker / Elasticsearch / SCE path is Old, not this product.