Where will every aircraft over Europe be in 60, 180 and 300 seconds, and is each one behaving the way aircraft behave here? Physics already answers the first question, unless the aircraft is doing something. The model predicts the residual over dead reckoning, speaks only when it disagrees with the straight line, and gets scored against reality when the truth lands a minute later: live, on the aircraft where it intervenes, it beats physics by ~20-30% on the same aircraft, on ~2,400 tracked aircraft, in public. The second question is answered by learned normalcy: occupancy profiles of the airspace itself, so the scope can flag the aircraft that is not doing what aircraft do there.
Trajectory. sky_trajectory, trained on self-labeled pairs from the live
stream (354k pairs, 4,031 aircraft, grouped 5-fold CV by flight,
absolute-error loss). CV, median horizontal error vs dead reckoning on the
maneuver-balanced training mix:
| horizon | physics (DR) | model | p90 physics | p90 model |
|---|---|---|---|---|
| 60 s | 406 m | 349 m | 3717 m | 3646 m |
| 180 s | 1728 m | 1445 m | 7211 m | 6454 m |
| 300 s | 2839 m | 2277 m | 15574 m | 12653 m |
Live, the model carries a deadband: it only overrides physics when it predicts the straight line will be off by more than 400/900/1500 m per horizon, because live traffic is cruise-dominated and DR in cruise is near-perfect. A residual model that always speaks loses (515 m vs 194 m); gated to where it disagrees, it wins: 964 m vs 1427 m median on the same aircraft (60 s, ~500 interventions / 5 min, morning wave). When physics is already right, the model stays silent. The model learned to shut up.
Ship rule, per horizon: beat physics on grouped-CV median AND p90, or that horizon stays physics. Retrains daily on the accumulating stream; a new version registers only if it beats the reigning champion in CV.
Normalcy. sky_normalcy: occupancy profiles per (1° cell, 45° track
sector, altitude band) built from the full state history: traffic share plus
ground-speed and vertical-rate quantile bands, capped per airframe so one
loiterer cannot define normal. Every aircraft gets a live anomaly score:
flying a flow that carries no traffic here (FLOW), or speed / vertical rate
outside the local band AND past a physical floor (GS / VS), so night-tight
quantiles cannot flag an ordinary climb. Scores and thresholds are evaluated
on a held-out 6 h before every ship; live: ~95% of airborne traffic scored,
a few percent flagged, each with its reason and the local expected bands on
the scope.
Read these before quoting any number anywhere.
- Dead reckoning at 60 s cruise is nearly unbeatable. Numbers are reported per phase and per horizon so the model cannot hide in cruise.
- Normalcy is occupancy, not intent. A rare-but-legitimate flight (survey, calibration, military exercise) scores anomalous; the scope shows why (flow share, expected bands) so a human can judge.
- The jamming grid is population-level. A hot cell means many aircraft report degraded nav integrity (low NIC share) there, not per-aircraft proof of interference.
- Labels are self-supervised from the stream. The true position arrives 60/180/300 s after each state: no annotation, millions of pairs per day, but also no labels for what the feed never saw.
- airplanes.live is community-run. The collector tolerates gaps and 429s; the feed-age indicator on the scope shows any hole.
An FTI (feature, training, inference) system on Hopsworks.
flowchart LR
A[airplanes.live\n8 tiles, 30s sweeps] -->|sky-collect\nbatched inserts| B[(aircraft_state\nonline + offline FG)]
B -->|sky-integrity, 5 min| C[(gps_integrity_cell FG)]
B -->|sky-pairs, hourly\nself-supervised labels| D[(trajectory_training FG)]
D --> E[trajectory_fv]
E -->|sky-train, daily\nGBM residuals vs physics\npromotion gate| F[[sky_trajectory]]
B -->|sky-normalcy, daily\noccupancy profiles| N[[sky_normalcy]]
F -->|KServe| G{{skywatch\ndeployment}}
N -->|KServe| G
B -->|RonDB key lookup| G
G -->|trajectories + anomaly scores| H[skylive app\ncommand center]
C --> H
The file-by-file map:
skywatch_features.py shared extraction: normalize, features, dead
reckoning, residual targets (no skew)
normalcy.py shared airspace profiles: binning, build,
anomaly score (no skew)
collect/sweep.py 8-tile Europe sweep, rate-limit aware
pipelines/feature_pipeline.py sweeps -> aircraft_state FG (Hopsworks job, 15 min)
pipelines/integrity_pipeline.py states -> gps_integrity_cell FG (Hopsworks job, 5 min)
pipelines/pairs_pipeline.py history -> multi-horizon pairs (Hopsworks job, hourly)
pipelines/train.py feature view -> model -> registry (Hopsworks job, daily)
pipelines/normalcy_pipeline.py history -> profiles -> registry (Hopsworks job, daily)
serving/ custom KServe predictor (RonDB lookups,
both bundles, deadband, anomaly score)
app/server.py command-center backend (poller, self-score,
interrogation, integrity layer)
app/static/ canvas radar client (no CDN, vendored Natural Earth)
app/deploy_app.py managed-app deploy, full recovery sequence
tools/schedule.py attach Quartz schedules to jobs
reqs/live-sky-watch.md the FTI specification
Clone into a Hopsworks project on the /hopsfs/... FUSE mount. Paths
self-derive; nothing is hardcoded to a username.
make collect-job # deploy + schedule the sweeper (Hopsworks job)
make integrity-job # deploy + schedule the jamming grid (Hopsworks job)
make pairs-job # deploy + schedule the pair builder (Hopsworks job)
make train-job # deploy + schedule the daily retrain (Hopsworks job)
make normalcy-job # deploy + schedule the profiles (Hopsworks job)
python3 serving/deploy_serving.py # KServe deployment
python3 app/deploy_app.py # the command center (Hopsworks app)
skylive: a command-center scope, not a dashboard. Three tabs, one question
each. Prediction: every aircraft altitude-colored (orange low, cyan
cruise), two futures per track (amber dot physics, cyan dot the model),
altitude filter chips, click a track for the spotlight: everything else
fades, the flown path draws itself, route / operator / airframe interrogated
live. Anomalies: normal traffic fades to dots, the ranked attention rail
explains each flagged aircraft in plain words (speed vs the local usual,
against-the-flow, emergency squawks); idle 45 s and the scope tours the list
itself. GPS integrity: the jamming grid with hot zones ranked. The top
strip is the honest part: physics vs model error on the same aircraft,
measured live, no cached demo numbers.