Control

Control

FastAPI next to Crawl. SQLite holds jobs, seeds, and labels. Pages live in Solr.

Jobs and seeds

Create a job, add seed URLs (or bin/sparkler inject). Each seed’s host is allowed when same host is on. Add a ucla.edu seed if you want that site in the same job without following the whole web.

Seeds are searchable in the View. GET /api/jobs/{id}/seeds?q= filters the list.

Labels and score

On Documents, mark pages not / relevant / highly. Control trains a keyword log-odds scorer from those labels (needs at least one relevant and one not). New and unfetched URLs pick up page_score; generate sorts by score then depth.

Start crawl

POST /api/jobs/{id}/crawl with {topn, iterations, same_host, max_depth}. iterations: -1 means until empty. The UI until done checkbox sends that. A crawl runs in a background thread; the bar shows the current URL and the header counts poll.

There is no Stop button yet — Clear is disabled while a crawl runs.

Reset

DELETE /api/jobs/{id}     # one job
DELETE /api/catalog       # every job + Solr crawldb

409 if a crawl is running.

API (selected)

GET  /api/health
GET  /api/jobs
GET  /api/jobs/{id}/stats
GET  /api/jobs/{id}/charts
GET  /api/jobs/{id}/documents
GET  /api/jobs/{id}/frontier
GET  /api/jobs/{id}/page?url=
GET  /api/jobs/{id}/seeds?q=
POST /api/jobs/{id}/seeds
POST /api/jobs/{id}/label
POST /api/jobs/{id}/crawl
DELETE /api/jobs/{id}
DELETE /api/catalog
GET  /api/export?job=&fmt=json