MatchVision AI: Sports Video Analytics & Tracking Pipeline

MatchVision AI converts broadcast football video into tracked detections, team assignments, real-world pitch coordinates, movement trails, distance estimates, and a synchronized 2D tactical view.

Click the thumbnail above to open the tactical-camera source video.

It shows the first three samples from the generated contact sheet. Each row pairs the tactical broadcast frame—YOLO boxes, team colors, track labels, and projected pitch markings—with the corresponding top-down 2D player locations.

The current pipeline combines:

• a fine-tuned YOLO detector for Player, Ball, and Referee;
• Ultralytics ByteTrack for persistent track IDs;
• jersey-color classification with temporal label stabilization;
• PnLCalib semantic pitch calibration every 30 frames;
• guarded optical flow between absolute calibration keyframes;
• image-to-pitch projection on a 105 m × 68 m field;
• a Streamlit dashboard for playback and frame-level inspection.

What has been implemented

Detection and ball-focused YOLO fine-tuning

The original class distribution was strongly imbalanced:

Class	Training boxes
Player	43,433
Ball	2,863
Referee	4,234

Ball annotations were only about 6.6% of the player count. A normal unbalanced training run was consequently dominated by the much larger and easier player boxes: player performance improved quickly, while the tiny ball remained easy for the loss to ignore and suffered poor recall.

The current fine-tuning pipeline addresses that failure mode with:

• a ball-focused training manifest that repeats/crops minority-heavy images;
• class-weighted classification loss [Player=0.45, Ball=2.50, Referee=1.20];
• 960-pixel training and inference for better small-object detail;
• low mosaic usage near the final epochs so tiny objects retain realistic scale;
• class-specific inference thresholds;
• tiled evaluation for checking whether additional local resolution helps the ball.

After balancing, the retained model reaches 0.681 ball mAP50, 0.655 recall, and 0.761 precision on the full-frame test set. Player mAP50 remains 0.951, so improving the minority class did not require sacrificing the main player detector.

Tracking and class stabilization

Ultralytics ByteTrack associates detections across frames. A track-level label smoother prevents a player from turning into a referee because of a single noisy prediction. Strong red/blue jersey evidence can also correct persistent player/referee confusion, while temporal team voting keeps team colors stable under lighting changes and partial occlusion.

PnLCalib, homography, and optical flow

PnLCalib detects semantic pitch keypoints and estimates an absolute camera-to-pitch homography. The accepted transform maps image pixels into a metric 105 m × 68 m pitch coordinate system.

The dashboard currently runs PnLCalib every 30 frames—roughly once per second for this source—and guarded optical flow fills the frames between those absolute anchors. Optical flow is used primarily to reduce calibration inference cost and latency; running the HRNet-based PnLCalib networks on every frame would be substantially slower. Because PnLCalib runs in a separate keyframe stage before YOLO cache generation, its GPU memory is also released before the main tracking pass.

Flow updates are accepted only when they:

• have sufficient forward/backward-consistent pitch features;
• pass RANSAC inlier checks;
• produce physically plausible pitch geometry;
• remain below the configured pitch-line alignment error;
• avoid implausibly large frame-to-frame projection changes.

Reliable PnLCalib anchors override optical flow, even when the matrices differ greatly. This is essential after camera pans, zooms, cuts, and long sequences where incremental motion would otherwise drift.

2D projection and analytics

For each tracked player or referee, the bottom-center of the bounding box is treated as the foot point. The image-to-pitch homography projects that point onto the 2D field. The ball uses its box center.

The generated cache and dashboard provide:

• synchronized video overlay and top-down 2D pitch;
• persistent track IDs and team-colored detections;
• selectable player trajectories over a configurable frame window;
• per-player distance covered in metres;
• instantaneous, average, and maximum speed estimates;
• per-team player counts and total distance;
• frame-by-frame projected coordinates;
• player/ball/referee mapping coverage;
• homography source, flow shift, and line-alignment diagnostics;
• a tactical reference-line overlay;
• browser playback plus exact frame navigation.

Distance accumulation rejects physically implausible jumps above the configured speed limit, reducing spikes caused by track switches or temporary homography failures.

Frame 7699: why absolute calibration matters

Click the comparison image to open the source video:

At frame 7699, long-running cumulative optical flow had a pitch-line alignment error of 43.29 px. A fresh PnLCalib estimate on the same frame reduced it to 9.07 px. The production pipeline therefore re-runs PnLCalib every 30 frames and uses guarded optical flow only for the short intervals between those anchors.

Current architecture

video frame
   │
   ├── YOLO detection ── ByteTrack ── label/team stabilization
   │
   └── PnLCalib keyframe ── guarded optical flow ── homography validation
                                      │
                                      ▼
                         player foot point → pitch metres
                                      │
                                      ▼
                         cache, metrics, video overlay, minimap

Important implementation files:

src/dashboard/pitch_tracking_app.py       Streamlit application
scripts/precompute_video_player_metrics.py Detection, tracking and projection cache
scripts/run_pnlcalib_image_folder.py      Absolute pitch calibration
scripts/evaluate_pipeline_cache.py        End-to-end cache diagnostics
scripts/compare_flow_pnlcalib_frame.py    Homography comparison utility
src/calibration/                          Pitch geometry and guarded motion tracking
src/training/                             YOLO training and evaluation

Requirements

• Python 3.11 or 3.12
• NVIDIA GPU recommended
• CUDA-compatible PyTorch
• FFmpeg for browser-compatible H.264 preview generation

Install:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

The repository keeps the required PnLCalib source under external/pnlcalib/. Its two calibration checkpoints exceed GitHub's standard per-file size limit and must be placed locally at:

external/pnlcalib/weights/SV_kp
external/pnlcalib/weights/SV_lines

The active YOLO checkpoint is included in Git because it is approximately 40 MB:

outputs/training_runs_soccana_ball_focus_v2/best/weights/best.pt

Required video type and input workflow

This project is designed for a tactical football camera: a high, wide broadcast view where a substantial portion of the pitch and its markings remain visible. The analytics are not intended to work reliably on every kind of football video.

Recommended input:

• elevated sideline or tactical broadcast camera;
• wide view containing multiple players and visible pitch geometry;
• mostly continuous camera motion;
• standard football pitch markings;
• limited close-ups, replays, crowd shots, graphics-only frames, and abrupt cuts.

Close-up player cameras, handheld footage, body cameras, highlight montages, heavily cropped social-media clips, and views with almost no visible lines cannot provide reliable PnLCalib homography or metric speed/distance estimates.

Put the tactical-camera video somewhere inside the local project, for example:

data/match_tactical.mp4

Video files are intentionally ignored by Git and are not uploaded to GitHub.

Run the dashboard

MPLCONFIGDIR=/tmp/matplotlib \
YOLO_CONFIG_DIR=/tmp/Ultralytics \
.venv/bin/streamlit run src/dashboard/pitch_tracking_app.py \
  --server.address 127.0.0.1 \
  --server.port 8501

Open http://127.0.0.1:8501.

In the dashboard sidebar:

1. enter the local tactical video path, such as data/match_tactical.mp4;
2. enter the absolute Start frame;
3. enter the absolute End frame;
4. choose detection size 960 for the trained model;
5. select Precompute Segment;
6. wait for PnLCalib keyframes, YOLO detection, ByteTrack, projection, cache generation, and preview encoding to complete.

The frame range is capped at 3,000 frames per run to keep processing and dashboard memory manageable. Generated caches and preview videos remain local under outputs/pitch_registration/ and are not committed to GitHub.

Precompute from the command line

The dashboard is the recommended workflow because it extracts the requested frame range and automatically generates PnLCalib anchors every 30 frames. The lower-level cache command expects an already generated PnLCalib JSON:

MPLCONFIGDIR=/tmp/matplotlib \
YOLO_CONFIG_DIR=/tmp/Ultralytics \
.venv/bin/python scripts/precompute_video_player_metrics.py \
  --video data/match_tactical.mp4 \
  --start-frame 5000 \
  --end-frame 7999 \
  --anchor-frame 5000 \
  --anchor-json path/to/pnlcalib/per_sample_output.json \
  --weights outputs/training_runs_soccana_ball_focus_v2/best/weights/best.pt \
  --device 0 \
  --imgsz 960 \
  --preview-fps 24 \
  --output-dir outputs/pitch_registration/my_segment

Model and dataset

The active detector is:

outputs/training_runs_soccana_ball_focus_v2/best/weights/best.pt

It was trained at imgsz=960 on the Soccana/SoccerNet-style dataset:

data/raw/soccana_v1/V1

Classes:

0 Player
1 Ball
2 Referee

Current full-frame test results:

Class	Precision	Recall	mAP50	mAP50–95
Player	0.924	0.915	0.951	0.629
Ball	0.761	0.655	0.681	0.368
Referee	0.864	0.833	0.873	0.549
Overall	0.850	0.801	0.835	0.515

The complete result is stored in outputs/evaluation/ball_focus_v2_app_fullframe_test.json.

Training

Continue fine-tuning the current checkpoint:

MPLCONFIGDIR=/tmp/matplotlib \
YOLO_CONFIG_DIR=/tmp/Ultralytics \
.venv/bin/python -m src.training.train_yolo \
  --config configs/training_soccana_ball_focus_best.yaml

Dataset preparation helpers:

./scripts/download_soccana_dataset.sh
./scripts/prepare_soccana_dataset.sh
./scripts/inspect_soccana_dataset.sh

Evaluation

1. Object detection

This is the ground-truth evaluation for detector accuracy:

MPLCONFIGDIR=/tmp/matplotlib \
YOLO_CONFIG_DIR=/tmp/Ultralytics \
.venv/bin/python -m src.training.evaluate \
  --model outputs/training_runs_soccana_ball_focus_v2/best/weights/best.pt \
  --data configs/soccana_ball_focus_best.yaml \
  --imgsz 960 \
  --device 0 \
  --batch 8 \
  --split test \
  --output outputs/evaluation/current_detection.json

Use src.training.evaluate_tiled when specifically investigating tiny-ball recall with tiled inference.

2. Homography and complete pipeline diagnostics

Evaluate a generated cache:

.venv/bin/python scripts/evaluate_pipeline_cache.py \
  --cache outputs/pitch_registration/video2_player_metrics_5000_7999_img960_fps24/analysis_cache.json \
  --output reports/current_pipeline_evaluation.json

The report measures:

• pitch-line alignment error: median, p95, maximum and threshold failures;
• percentage of detections successfully mapped onto the pitch;
• label stability between player and referee;
• track-gap and continuity proxies;
• YOLO inference latency;
• absolute-anchor and optical-flow usage.

Current 3,000-frame cache:

Diagnostic	Result
Mapping coverage	99.55%
Median line error	7.43 px
P95 line error	13.08 px
Frames above 18 px	7 / 3000
Median YOLO latency	20.94 ms
P95 YOLO latency	22.68 ms

3. Proper tracking evaluation

The cache diagnostics are useful but are not HOTA, MOTA, or IDF1. Proper multi-object tracking evaluation requires frame-level ground-truth boxes and persistent identity IDs for the same video.

For a publishable tracking evaluation:

1. annotate or obtain ground-truth tracks for representative clips;
2. export predictions and ground truth in MOTChallenge format;
3. evaluate with TrackEval;
4. report HOTA, DetA, AssA, IDF1, MOTA, ID switches, and fragmentation;
5. evaluate player, referee, and ball tracks separately.

This distinction matters: high detector mAP does not guarantee stable identities, and a visually plausible minimap does not guarantee an accurate homography.

Tests

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .venv/bin/pytest -q

Docker

Docker configuration is provided for future deployment, but containerization is optional and is not required for current local development. Building the image duplicates dependencies and model files, so skip it when disk space is limited.

When sufficient storage is available:

docker compose up --build

The dashboard is exposed on port 8501.

Output policy

Generated caches, previews, local videos, runtime configuration files, calibration keyframes, and temporary evaluation products are ignored by Git. The repository keeps source code, documentation images, compact evaluation reports, and the active YOLO checkpoint.

License

MatchVision AI is released under the MIT License. Vendored PnLCalib code remains subject to its own upstream license in external/pnlcalib/LICENSE.