DataHub Agent — verifier-gated web-scraping agent that catalogs results into DataHub

Agent Hackathon 2026 — "Agents That Do Real Work"

What it does

An autonomous agent that:

1. Scrapes the web with a headless browser (JS-rendered pages included) → clean JSONL
2. Writes metadata into DataHub: registers datasets + schemas + dataJob runs + lineage + row-count/freshness assertions — the agent contributes back to the graph, it doesn't just read it
3. Acts on DataHub context: before each run it reads what's already connected, prior run receipts, and schemas; results are linked to previous runs via lineage
4. Is verifier-gated: the agent may only write to the graph when the scrape gate passes (rows > 0) — no silent garbage ingestion

Why it's different

• Scrapers are usually scripts; this one is an agent with a hard verifier gate (the same discipline used in production autonomous coding agents)
• Every run leaves durable lineage + assertions in DataHub, so the next agent (or human) inherits context instead of starting from zero
• Zero-infra demo mode: run without DataHub and it writes the exact same receipts (catalog/run-*.json) — then point it at a real DataHub via DATAHUB_GMS_URL and the same code ingests via the OpenAPI entities endpoint

Quick start (offline demo — 2 minutes)

pip install -r requirements.txt
python3 datahub_agent.py --target https://news.ycombinator.com/
# -> [gate] scrape produced 30 rows
# -> [offline] wrote catalog/run-....json

Quick start (real DataHub)

# DataHub quickstart (docker compose, ~6GB RAM): see datahubproject.io/docs
export DATAHUB_GMS_URL=http://localhost:8080
export DATAHUB_TOKEN=<your token>
python3 datahub_agent.py --target https://news.ycombinator.com/

Built with

• The Web Scraper Starter Kit (headless browser → JSONL → DuckDB)
• DataHub OpenAPI entities endpoints
• Verifier-gated agent loop (build/test/invariant gates before any write)

Architecture

target site → headless browser → HTML → parser → JSONL
                                              │  (verifier gate: rows>0)
                                              ▼
                                     DataHub (dataset, schema,
                                     dataJob, lineage, assertions)
                                              ▲
                     next run reads context ←──┘ (agent acts on the graph)