Getting Started

Getting started

Needs Java 21+ (Solr 10), Python 3.10+, and Node 18+ to build the UI. Tika can use Java 11+; Sparkler points Solr at Java 21 when it finds Homebrew openjdk@21.

git clone https://github.com/USCDataScience/sparkler.git
cd sparkler
python3.12 -m venv .venv
.venv/bin/pip install -e .
cd web && npm install && npm run build && cd ..

bin/sparkler inject -id demo -sf demo/seeds.txt
bin/sparkler crawl  -id demo --same-host -i -1
bin/sparkler serve

Open http://127.0.0.1:8180/

bin/sparkler uses .venv when it exists.

Solr 10 slim is downloaded on first use into data/ (gitignored) and bound to 127.0.0.1:8983. The app is 127.0.0.1:8180. Those ports are chosen so they do not collide with DRAT (9000), ImageCat live (9100), BigTranslate (9200 / Solr 8985), or Meridian (8090).

bin/sparkler inject -id news -su 'https://example.com/'
bin/sparkler crawl  -id news --same-host -i -1
bin/sparkler reset  -id news --yes
bin/sparkler solr   status

Crawl a whole site (UI)

1. Create a job, add a seed (e.g. https://mattmann.ai/).
2. same host on — www and apex count as one site.
3. until done on — do not stop after one batch of 50.
4. depth blank or −1 — no hop cap on that site.
5. Crawl. The header counts refresh while it runs. You want frontier 0.

Off-site outlinks are not fetched. When same-host finishes, leftover off-site UNFETCHED URLs are marked FILTERED so the frontier can empty.

Hover same host, depth, and until done for tooltips.

Add extra seeds (a ucla.edu URL) if you want those hosts included while same host stays on. Uncheck same host only if you intend to follow off-site links, and set a depth (2 or 3) so it does not wander.

Clear the catalog

In the bar: Clear job (this job) or Clear all (every job + Solr). Confirms first. Disabled while a crawl is running.

bin/sparkler reset -id mai-job --yes
bin/sparkler reset --all --yes

The 2016–2023 Spark / Docker / Elasticsearch / SCE path is Old, not this product.