ScopePilot

Local-first AI endoscopy workflow assistant — research/hackathon prototype.

ScopePilot ingests recorded or de-identified endoscopy video and turns it into structured, reviewable procedural data: lesion detections, anatomical localization, biopsy tracking, an event timeline, quality metrics, and an AI-assisted draft report — with the physician confirming or dismissing every AI output.

⚠️ Not for clinical use. ScopePilot is not a medical device, not clinically validated, not HIPAA compliant, and not FDA approved. It must never be used for diagnosis, treatment, or real patient care. Use only public, synthetic, recorded, or de-identified demo data — never real PHI.

Architecture

┌──────────────────────────── Next.js frontend (localhost:3000) ────────────────────────────┐
│ login · procedure list · upload · LIVE DASHBOARD (frames + bbox overlay, segment,         │
│ timers, biopsy counters, confirm/dismiss) · timeline · report editor · export · admin     │
└───────────────▲──────────────────────────────────────────────▲───────────────────────────┘
                │ REST (same-origin session cookie)             │ WebSocket (single-use ticket)
┌───────────────┴──────────────────────────────────────────────┴───────────────────────────┐
│ FastAPI backend (localhost:8000)                                                          │
│ session auth · RBAC · procedures · video service (OpenCV sampling) · inference service    │
│ (YOLO lesion + instrument, timm anatomy — placeholders until weights exist) · temporal    │
│ tracker (IoU + smoothing) · procedure state manager · biopsy service · screenshots ·      │
│ quality metrics · report engine · de-identified export (deid guard) · audit log           │
└───────────────┬───────────────────────────────────────────────────────────────────────────┘
                │ SQLite (users, procedures, events, reports, audit) + local filesystem
                │ (videos, frames, screenshots, exports) — everything stays on this machine

Details: docs/ARCHITECTURE.md, docs/API.md, docs/DATA_MODEL.md, docs/RBAC.md, docs/EVENT_FLOW.md, docs/TRAINING.md, docs/SECURITY.md.

Setup

Backend (Python 3.9+):

cd backend
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
cp .env.example .env                       # edit SECRET_KEY etc.
.venv/bin/alembic upgrade head
.venv/bin/uvicorn app.main:app --reload --port 8000 --env-file .env

Database schema changes are applied through Alembic, not application startup. For an existing pre-Alembic database, back it up, then run this once after installing dependencies:

cd backend
.venv/bin/alembic stamp 0001

Frontend (Node 18+):

cd frontend
npm install
cp .env.local.example .env.local           # optional: direct API/WS URLs for localhost:8000
npm run dev                                # http://localhost:3000

Large video uploads post directly to the FastAPI backend (NEXT_PUBLIC_API_DIRECT_URL, defaulting to the current browser host on port 8000) instead of going through the Next.js rewrite proxy. This avoids Next's 10 MB proxy body buffer and is faster for local video files.

Environment variables (backend/.env.example)

Variable	Default	Purpose
SECRET_KEY	dev value (change it)	Session hardening guard; change outside demo mode
DEMO_PASSWORD	demo1234	seeded demo accounts
FRONTEND_ORIGIN	http://localhost:3000	CORS allowlist
DATA_DIR	./data	videos/frames/screenshots/exports/SQLite
MAX_UPLOAD_MB	500	upload size limit
FRAME_SAMPLE_FPS	5.0	processing sample rate
PROCESS_REALTIME	true	pace processing for a live demo (false = as fast as possible)
INSTRUMENT_CONFIDENCE_THRESHOLD	0.75	minimum forceps detection confidence
INSTRUMENT_MIN_TRACK_HITS	5	consecutive samples required for an instrument event
LESION_CONFIDENCE_THRESHOLD	0.35	minimum lesion YOLO detection confidence
DEMO_MODE	true	false refuses dev secrets and skips demo-user seeding
COOKIE_SECURE	false	set true when serving over HTTPS (e.g. mkcert local TLS)
LESION_MODEL_PATH / INSTRUMENT_MODEL_PATH / ANATOMY_MODEL_PATH	models/weights/*.pt	real weights; placeholders used when missing

Demo walkthrough

1. Generate a fully synthetic demo video (no real footage): backend/.venv/bin/python scripts/make_demo_video.py data/videos/demo.mp4 90
2. Log in at http://localhost:3000 as physician@demo.local / demo1234.
3. New procedure → upload the demo video → Start analysis.
4. Watch the live dashboard: frames, tracked detections, anatomical segment (override anytime), timers, quality indicator. Log biopsies with the buttons or keys B/T; screenshot with S.
5. Confirm/dismiss AI detections (dashboard or timeline), start withdrawal, stop processing.
6. Generate the draft report (clearly labeled AI-generated), edit it, then build a de-identified export and download the bundle.

Demo accounts (all demo1234): admin@ (everything) and physician@ (full clinical workflow). RBAC is enforced in backend route dependencies — the UI merely hides what a role cannot do. All 401/403s, logins, uploads, edits, and exports are written to the audit log (admin → Admin page).

Local inference & training

The app never depends on training code — it loads weights from models/weights/ via env paths and falls back to deterministic placeholder models with identical output schemas, so the whole workflow demos without any trained model. To train real models (see docs/TRAINING.md):

• Lesion/polyp (YOLO): Kvasir-SEG masks → scripts/convert_kvasir_seg_to_yolo.py → yolo detect train … → models/weights/lesion_yolo.pt (guide)
• Instrument/forceps (YOLO): Kvasir-Instrument → scripts/convert_kvasir_instrument_to_yolo.py (guide)
• Anatomy (timm ConvNeXt-Tiny): HyperKvasir → scripts/prepare_hyperkvasir_anatomy.py → training/anatomical_localization/train.py (guide)

Datasets, weights, videos, frames, exports, and databases are gitignored. ONNX Runtime / TensorRT are planned as swappable inference backends behind the same interfaces.

De-identification

Exports contain only synthetic case ids (CASE-0001), video-relative timestamps, structured events, metrics, report text, and annotated screenshots. A de-identification guard (backend/app/services/deid.py) strips any key whose name suggests an identifier (name, MRN, DOB, facility, clinician, contact info) plus wall-clock dates before anything is written to an export. Raw video and frames never leave the machine, and nothing is sent to external APIs.

Local TLS (optional, recommended on a LAN)

Everything stays local; TLS just protects traffic between procedure-room machines and this server and lets the session cookie use Secure. Using mkcert:

mkcert -install                                   # one-time local CA
mkcert localhost 127.0.0.1 <lan-ip>               # emits ./localhost+2.pem + key
cd backend && .venv/bin/uvicorn app.main:app --port 8000 --env-file .env \
  --ssl-certfile ../localhost+2.pem --ssl-keyfile ../localhost+2-key.pem

Then set COOKIE_SECURE=true in backend/.env, use https:// in FRONTEND_ORIGIN and set NEXT_PUBLIC_WS_URL=wss://<host>:8000 for the frontend if needed. Trust the mkcert CA on any LAN client machines (mkcert -CAROOT shows the CA to copy). Certificates and keys are gitignored artifacts — never commit them.

Security notes (HIPAA-aware, not HIPAA-compliant)

Opaque revocable session auth (httpOnly cookie or bearer; single-use WebSocket tickets; login rate limiting; CSRF header check on cookie-authenticated writes) · backend-enforced RBAC · upload type/size/magic-byte validation · server-generated file names + path-traversal guards on all media routes · restricted CORS · secrets via env vars · audit logging without clinical content · deletable procedures with artifact cleanup. See docs/SECURITY.md.

Tests

cd backend && .venv/bin/python -m pytest tests -q     # API, RBAC, deid, tracking, WS
backend/.venv/bin/python scripts/tests_smoke.py       # dataset conversion helpers
cd frontend && npm run build                          # type-checked production build

Limitations

Placeholder CV models by default (synthetic detections); anatomy estimation is heuristic until a classifier is trained; single-process demo server (no horizontal scaling); SQLite storage without encryption at rest; local session auth rather than a hosted IdP; no clinical validation of any output.

Roadmap

Trained YOLO/ConvNeXt weights · ByteTrack upgrade · ONNX/TensorRT backends · local LLM (e.g. Gemma) report drafting from the de-identified event JSON · Postgres + encrypted object storage · EMR/reporting integration mockup · live scope-stream ingestion.