OffSight

Explainable AI football referee. OffSight analyses match clips and incident data to deliver auditable offside and foul verdicts — deterministic geometry computes the objective facts, a multi-agent Claude pipeline constructs the legal reasoning, and a consistency guard ensures the AI can never contradict what the geometry already proved.

---

What it does

Football decisions affect outcomes yet the reasoning behind them is almost never explained in full. OffSight addresses this for two common incident types:

• Offside — computes the second-last-defender line per Law 11, identifies which attackers are beyond it and by how much, then reasons through the full rule (active play, deliberate save, interfering with an opponent) to reach a verdict with explicit justification.
• Foul — analyses proximity, velocity differentials, and contact geometry to assess whether a challenge was reckless, excessive, or careless.

The system can work from pre-loaded demo cases (instant) or from uploaded broadcast video (slow — the CV pipeline tracks players from real video and builds a case from scratch).

---

Architecture

┌─────────────────────────────────────────────────────────────┐
│ Layer 1 — Demo case library (pre-computed JSON)             │
│   Instant results from carefully annotated incidents        │
└──────────────────────────┬──────────────────────────────────┘
                           │ or                               
┌──────────────────────────▼──────────────────────────────────┐
│ CV Pipeline (optional, slow)                                │
│   Broadcast .mp4 → SoccerNet GSR (sn-gamestate + TrackLab) │
│   → per-frame minimap positions → case JSON                 │
└──────────────────────────┬──────────────────────────────────┘
                           │
┌──────────────────────────▼──────────────────────────────────┐
│ Layer 2 — Deterministic geometry engine (pure JS)           │
│   Law 11 offside line, defender count, beyondBy metres      │
│   No LLM involved — objective, reproducible                 │
└──────────────────────────┬──────────────────────────────────┘
                           │
┌──────────────────────────▼──────────────────────────────────┐
│ Layer 3 — Multi-agent Claude pipeline (5 agents)            │
│   fact-extraction → rule-retrieval → verdict                │
│                 → challenge → synthesis                     │
│   Each agent has a narrow role; degraded gracefully         │
└──────────────────────────┬──────────────────────────────────┘
                           │
┌──────────────────────────▼──────────────────────────────────┐
│ Layer 4 — Consistency guard (deterministic override)        │
│   If geometry proves X, the final verdict must say X.       │
│   Prevents LLM from hallucinating a contradictory result    │
└─────────────────────────────────────────────────────────────┘

The consistency guard is the critical piece: if the geometry shows a player is 0.8 m offside, the guard will override an LLM verdict that says "not offside" with the correct geometric conclusion, annotating the result as overridden. The LLM reasoning chain is still shown — only the headline verdict is corrected.

---

Demo

The app ships with pre-computed demo cases covering several offside scenarios and a foul. Select a case on the landing page, then click Analyse to run the full reasoning pipeline.

Video upload requires either a local CV environment (see below) or a Colab GPU worker pointed at the backend via CV_WORKER_URL. For quick demos, the pre-loaded cases are indistinguishable from real video uploads in terms of the reasoning output.

---

Getting Started

Prerequisites

• Node.js 20+ (backend + frontend build)
• API key: Claude (Anthropic) or Groq — one is required for Layer 3 reasoning
• Python 3.9 + uv (only if using video upload locally)
• ffmpeg (only if using video upload)

Main app

# 1. Clone
git clone https://github.com/amirjon-1/Offsight.git
cd Offsight

# 2. Backend
cd offsight/backend
cp .env.example .env
# Edit .env: set LLM_PROVIDER, the matching API key, and ANTHROPIC_MODEL/GROQ_MODEL
npm install
npm start          # listens on PORT (default 3001)

# 3. Frontend (separate terminal)
cd offsight/frontend
npm install
npm run dev        # Vite dev server, default :5173
# or: npm run build && npm run preview

Open http://localhost:5173. The landing page shows demo cases; pick one and hit Analyse.

Video upload — local CV pipeline

The CV pipeline requires Python 3.9 (exact), uv, ffmpeg, and enough CPU/RAM to run neural inference (~5 min per clip on a modern CPU).

# From repo root — uv downloads Python 3.9 automatically if needed
cd vision/sn-gamestate
uv venv --python 3.9
source .venv/bin/activate

# Install CUDA torch first if you have a GPU (adjust cu117/cu118 to match your driver):
# pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117

# CPU-only (default):
pip install torch==1.13.1 torchvision==0.14.1

pip install openmim==0.3.9
mim install "mmcv==2.1.0"
pip install -e ../tracklab
pip install -e plugins/calibration
pip install -e .

# Apply OffSight config overlays (custom pipeline config without jersey detection):
cp ../../vision-config/sn-gamestate/sn_gamestate/configs/customvideo.yaml \
   sn_gamestate/configs/customvideo.yaml
cp ../../vision-config/sn-gamestate/sn_gamestate/configs/dataset/uploadvideo.yaml \
   sn_gamestate/configs/dataset/uploadvideo.yaml
cp ../../vision-config/sn-gamestate/sn_gamestate/configs/modules/tracklet_agg/voting_role.yaml \
   sn_gamestate/configs/modules/tracklet_agg/voting_role.yaml
cp ../../vision-config/sn-gamestate/sn_gamestate/calibration/nbjw_calib.py \
   sn_gamestate/calibration/nbjw_calib.py

With the venv set up, the video upload feature in the frontend will work automatically (the backend spawns scripts/run_gsr.sh which activates the venv internally).

Upload tips: 720p+ broadcast-style clips, ~10–60 seconds, wide field view. Tight angles and low-resolution clips fail player detection.

Video upload — Colab GPU worker

For hosted deployments (Render, Railway, Fly.io) where running neural inference on the server is impractical, a Colab notebook sets up the CV pipeline on a free T4 GPU and exposes it via ngrok.

Open colab/offsight_cv_worker.ipynb in Google Colab:

1. Runtime → Change runtime type → T4 GPU
2. Run all cells (Ctrl+F9) — first run takes ~15 min; subsequent runs ~3 min if you connected Google Drive in the Drive cell
3. Copy the ngrok URL printed by the last setup cell
4. Set CV_WORKER_URL=<that URL> in your backend .env or hosting dashboard environment variables
5. Restart the backend — video uploads will now proxy to Colab instead of running locally

Per-session restart (ngrok free tier gives 8-hour tunnels):

1. Open the notebook in Colab
2. Runtime → Run all
3. Copy the new ngrok URL from the output
4. Update CV_WORKER_URL in your backend
5. Restart the backend process

Model weights (~1.5 GB) download on first run. Mounting Google Drive in the Drive cell persists them so subsequent sessions skip the download.

---

Project structure

offsight/
  backend/         Express API — agent pipeline, geometry engine, case store
    src/
      agents/      Multi-agent Claude/Groq pipeline (pipeline.js)
      geometry/    Law 11 implementation, proximity/velocity analysis
      routes/      ingestVideo.js (CV job management), analyze.js, cases.js
  frontend/        Vite + React — case selector, minimap renderer, reasoning panel

scripts/
  run_gsr.sh       CV pipeline entrypoint: video → pklz tracker state
  pklz_to_case.py  Converts TrackLab state to OffSight case JSON
  sample_frames.py Frame sampling utility (window/stride modes)

vision/            (gitignored — clone fresh per machine)
  sn-gamestate/    SoccerNet GSR fork
  tracklab/        TrackLab upstream

vision-config/     Custom config overlays for sn-gamestate (tracked in git)
  sn-gamestate/
    sn_gamestate/configs/customvideo.yaml       Pipeline without jersey detection
    sn_gamestate/configs/dataset/uploadvideo.yaml
    sn_gamestate/configs/modules/tracklet_agg/voting_role.yaml
    sn_gamestate/calibration/nbjw_calib.py

colab/
  offsight_cv_worker.ipynb   Colab notebook for GPU-accelerated CV worker

---

Built With & Attribution

Core frameworks

Tool	Use
Express.js	Backend API server
React	Frontend UI
Vite	Frontend build tool and dev server
Node.js	Runtime (requires v20+)

CV pipeline

Tool	Use
sn-gamestate	SoccerNet Game State Recognition — the full tracking + calibration + role/jersey pipeline that converts broadcast video to minimap positions. Used as the core CV engine.
TrackLab	Modular tracking framework that sn-gamestate is built on. Provides the plugin architecture, Hydra config system, and .pklz state serialisation.
PyTorch 1.13.1	Deep learning runtime for all CV models (detection, re-ID, calibration)
OpenMMLab / mmdet / mmocr	Object detection and OCR components used by sn-gamestate (jersey number detection)
Hydra	Configuration management for the TrackLab/sn-gamestate pipeline
ffmpeg	Video decoding and frame extraction

AI / reasoning

Tool	Use
Anthropic Claude API	Primary LLM for the 5-agent reasoning pipeline (fact-extraction → rule-retrieval → verdict → challenge → synthesis)
Groq API	Alternative fast LLM backend (llama-3.3-70b-versatile); used for development and cost-efficient deployments

Data

Source	Use
SoccerNet Dataset	The sn-gamestate models were trained on SoccerNet tracking, calibration, and game-state annotation data. Pre-trained weights are downloaded automatically by TrackLab on first pipeline run.
Demo case JSON files	Manually constructed from annotated broadcast frames to represent clear offside/foul scenarios. No real player data.

Infrastructure & tooling

Tool	Use
uv	Python package manager for the CV pipeline venv; handles Python 3.9 constraint automatically
Google Colab	Free/Pro GPU environment for running the CV pipeline in hosted deployments
ngrok	Tunnel to expose the Colab CV worker as a public HTTPS endpoint
pyngrok	Python ngrok client used in the Colab notebook

Python libraries (CV pipeline)

Library	Use
prtreid	Person re-identification backbone (part-based) used by sn-gamestate
bpbreid / torchreid	Body-part-based re-identification models for robust player identity tracking across occlusions
nbjw-calib	Pitch calibration module (homography estimation from line markings) — bundled as the tracklab_calibration plugin
NumPy, OpenCV, Pandas	Standard CV / data processing stack

---

Notes on the CV pipeline

The vision/sn-gamestate and vision/tracklab directories are intentionally not committed to this repo (see .gitignore). They are large, have their own git histories, and must be cloned from upstream. The four files in vision-config/ are the only changes OffSight makes to the upstream pipeline:

• customvideo.yaml — removes jersey number detection (requires mmocr, fails on Apple Silicon; irrelevant to offside geometry) and sets the experiment name
• uploadvideo.yaml — dataset config for an arbitrary external video file
• voting_role.yaml — majority-vote tracklet aggregation for player role (goalkeeper / field player)
• nbjw_calib.py — patched calibration module that imports correctly from the installed tracklab_calibration package path

---

License

MIT