Catalog

The crawl graph is Solr core crawldb, not SQLite and not Elasticsearch.

id = sha1(crawl_id + "\n" + url) so the same URL can exist in two jobs.

Field	Role
`url`, `crawl_id`, `hostname`, `group`	Identity. `group` is the apex host (`www` stripped)
`status`	UNFETCHED, FETCHED, ERROR, FILTERED
`discover_depth`	Hops from a seed (pagination query strings still increment this)
`page_score`	Label scorer (or 0)
`title`, `extracted_text`, `content_type`	Tika / HTML
`tika_metadata`	Full Tika map as JSON — open a Documents row to see it
`*_s_md`	Flattened Tika keys, searchable (classic Sparkler convention)
`outlinks`, `parent`, `seed`	Graph
`contenthash`	SHA-1 of the fetched body
`label`	not / relevant / highly
`fetch_timestamp`, `response_time`	Used by the Stats heatmap and latency chart

Browse in the UI (Documents, click a row) or:

http://127.0.0.1:8983/solr/crawldb/select?q=crawl_id:mai-job&wt=json

GET /api/jobs/{id}/charts rolls FETCHED docs into the heatmap, donuts, bubbles, and treemap.

Jobs, seeds, and labels also live in data/sparkler.sqlite. Reset drops Solr docs for that job and the SQLite rows.

Inspired by USC IRDS Sparkler.

Old (2016–2023)

Navigation Menu