Sentinel — OSINT Aggregation Dashboard

Sentinel pulls together publicly available information about a domain or IP into one dashboard: exposed ports/services (Shodan), certificate transparency data and subdomains (crt.sh), domain registration data (WHOIS), and email addresses published on the target's own public web pages. Results are stored in SQLite so you can track how a target's exposure changes over time, with Chart.js trend lines on the target page.

Built with FastAPI (async, so the four lookups run concurrently instead of one after another) + SQLAlchemy + SQLite + server-rendered Jinja2 templates.

What it checks

Source	What it gathers	API key needed?
Shodan	Open ports, services, banners on the resolved IP	Optional — uses free InternetDB (ports only) without a key; add SHODAN_API_KEY for full banners/products/versions via the Host API
crt.sh	Certificates issued for the domain, and subdomains extracted from their SANs	No
WHOIS	Registrar, org, creation/expiry dates, name servers	No
Public pages	Emails found in mailto: links or visible text on the domain's own homepage/contact/about/privacy pages	No

The email harvester is intentionally passive — it only fetches pages the target already serves publicly (no guessing subdomains, no third-party data brokers, no active scanning).

Setup

Sentinel targets Python 3.14. A couple of things are worth knowing if you're on Windows with a fresh 3.14 install:

• Always install into a virtual environment, and always invoke pip/python the same way (python -m pip ..., then python run.py) — mixing a bare pip install with a differently-resolved python/uvicorn on PATH is the most common cause of ModuleNotFoundError: No module named 'uvicorn' after a seemingly successful install.
• requirements.txt deliberately leaves pydantic/pydantic-settings unpinned (>=2.10 / >=2.7) rather than locked to an exact version. Python 3.14 is new enough that older pinned releases of pydantic-core (its Rust-based validation engine) don't have prebuilt Windows wheels for it, which forces pip to compile from source — and that fails unless you have the MSVC C++ build tools installed (link.exe not found). Letting pip resolve to the latest pydantic-core avoids the compile step entirely, since current releases ship prebuilt cp314 wheels. Don't re-pin these to old exact versions without checking Python 3.14 wheel coverage first.
• If you ever do hit a wheel-build error on some other dependency, the fix is almost always either (a) bump that package's version the same way, or (b) fall back to Python 3.12, which has full wheel coverage everywhere.

cd sentinel
pip install -r requirements.txt

Run

python run.py

Then open http://127.0.0.1:8000.

• Add a target (domain like example.com, or a raw IP).
• Click Run New Scan on the target page. It queries all four sources concurrently, saves the results, and shows them on the dashboard.
• Run the scan again later on the same target to build up trend history — the chart on the target page plots ports/subdomains/certificates/emails found per scan over time.
• Delete Target (on the target list or the target detail page) asks you to type the target's name to confirm before it removes the target and its full scan history — there's no undo, so this is deliberately harder to trigger by accident than a plain "are you sure?" popup.

Interactive API docs (auto-generated by FastAPI) are at /docs.

Project layout

sentinel/
  app/
    main.py               FastAPI app + startup (creates SQLite tables)
    config.py              Settings loaded from .env
    database.py            SQLAlchemy engine/session
    models.py               Target, Scan, PortResult, SubdomainResult,
                             CertificateResult, EmailResult, WhoisResult
    schemas.py              Pydantic response models
    services/
      shodan_service.py     InternetDB / Shodan Host API
      crtsh_service.py      Certificate transparency lookups
      whois_service.py      WHOIS (blocking, run in a thread pool)
      email_service.py      Passive email harvesting from the target's own site
      aggregator.py          Runs all four concurrently, persists to DB
    routers/
      targets.py             CRUD for targets
      scans.py                Trigger scans, fetch scan history, trend data
      dashboard.py             Server-rendered HTML pages
    templates/               Jinja2 templates (dark navy/gold theme)
    static/                  CSS + (Chart.js loaded from CDN)
run.py                       Dev server entrypoint (uvicorn, --reload)

Notes on data sources

• crt.sh is a free, community-run service and is the flakiest of the four sources — it can time out or return unexpected error codes (404, 502, 504) under load, especially for domains with huge certificate histories (e.g. large, well-known sites). Sentinel retries once with a longer timeout and treats a 404 as "no matches" rather than a hard error; a persistent failure still gets recorded in the scan's Notes section without failing the whole scan.
• WHOIS is a blocking socket call (not HTTP), so it's dispatched via asyncio.to_thread alongside the other async lookups rather than blocking the event loop. Many gTLDs now redact registrant emails/names by default (GDPR-style privacy), so raw_emails will often be empty — that's expected, not a bug.
• Shodan without a key uses the free InternetDB endpoint, which only returns port numbers/CPEs, not banners. Add a key in .env for full service banners, products, and versions.
• Partial failures (one source down, rate-limited, etc.) don't fail the whole scan — each source's error is collected and shown in a "Notes" section on the target page, and whatever data did come back is still saved.