Track any object in a video and measure its real-world speed and acceleration — from a single camera, no special equipment.
Draw a box around an object in one frame. VeloCV segments and tracks it through the rest of the clip, reconstructs its actual 3D motion (not just pixel motion), and overlays live speed and acceleration on the video. For objects in free flight (a thrown or dropped ball), it fits a real physics model to the trajectory and reports how closely the recovered gravity matches the true value — 9.81 m/s² — as a built-in accuracy check.
Real footage, not synthetic: a ball tossed in a hallway, tracked and fit to a physics model in real time. The "9.81 m/s²" on screen is the fixed physics assumption used to compute the trajectory, not a happy coincidence — the actual measured/validated result for this clip is reported below.
video ──┬─> user draws a box around the object on one frame
└─> a YOLO pass suggests the category (person/car/ball/…) and a
sensible pose-stable dimension + default size to confirm
│
v
SAM2 (Segment Anything 2) segments and tracks the object's exact
mask through every frame, forward and backward from the prompt
│
v
each frame: mask → sub-pixel centroid + pose-stable dimension
(diameter for a ball, height for a person/car — whichever doesn't
change as the object rotates)
│
v
pinhole camera model (checkerboard-calibrated, or a documented lower-
accuracy "estimated" fallback) back-projects 2D pixels to 3D position
│
┌─────────┴─────────┐
v v
free-flight segment otherwise: per-frame position,
detected? fit a 6- Savitzky-Golay smoothed →
parameter ballistic velocity, acceleration
model (gravity fixes
absolute scale) →
launch velocity,
fitted g vs 9.81 m/s²
│ │
└─────────┬──────────┘
v
annotated output video + metrics.json + plots
Why this is genuinely 3D, not just pixel tracking: a camera flattens depth. VeloCV recovers it two ways. For an object in free flight, gravity's known, universal curvature pins down absolute scale from the 2D track alone — no object size needed. Otherwise, the object's own known real-world size (diameter/height) combined with how large it appears in each frame gives a depth estimate that updates continuously, which is what lets the tool handle an object moving toward or away from the camera, not just side-to-side.
Why mass isn't estimated: under ideal projectile motion, trajectory alone cannot recover mass — a bowling ball and a tennis ball thrown identically follow the exact same path (Galileo's equivalence principle). Claiming otherwise would be the kind of thing that falls apart under a follow-up question, so it's left out.
No synthetic dataset substitutes for this: since gravity is a known, universal constant, a real recorded throw or drop is its own ground truth. Four clips were recorded on an iPhone (checkerboard-calibrated, 0.64px reprojection error — see Calibration), all analyzed by the exact same pipeline used above.
| Clip | Mode | Result | Error vs. 9.81 m/s² |
|---|---|---|---|
| semiarc_toss | Flight (ballistic fit) | Fitted g = 8.95 m/s², launch speed 3.22 m/s, 1.42px reprojection residual | 8.72% |
| ball_drop | General (per-frame) | Mean acceleration 9.03 m/s² (median 8.80) | 7.91% (mean) / 10.3% (median) |
| straight_toss | General (per-frame) | Correct free-flight speed curve (peak-to-peak 5.03 → 0.31 → 4.51 m/s) | no independent ground truth for this one |
| arc_toss | General (per-frame) | Correct free-flight speed curve (4.75 → 0.74 → 4.17 m/s) | no independent ground truth for this one |
The two flight-type measurements land at ~8-9% error recovering a real physical constant — a legitimate, unglamorized number: a bit above the project's best-case theoretical estimate (2-5%, under ideal conditions), which is exactly what real footage (real motion blur, a hand-thrown toss, an imperfectly-measured object) should be expected to look like versus a lab-perfect synthetic test.
straight_toss and arc_toss don't have their own numeric ground truth (no
tape-measure/stopwatch check was done for those two), but both show the
correct physical shape — speed decreasing to a minimum at the arc's apex
and increasing symmetrically back down — which is a real, if qualitative,
correctness check.
Flight-mode residual fit (semiarc_toss) — how well the fitted 3D trajectory reprojects back onto the observed 2D track:
Velocity over time (semiarc_toss, flight mode) and (ball_drop, general mode):
Two real things this validation run surfaced (documented, not hidden)
- Phone autofocus hunting mid-recording tripped the camera-motion safety
check. The pipeline has a guard that refuses to output real-world
units if it detects the camera moving. Autofocus refocusing caused
enough temporary blur/optical shift to trigger it on the
ball_dropclip. Verified independently — tracked a fixed doorframe feature across the clip; it returns to within 1px of its starting position by the last frame, which real camera movement would not do — and overrode the guard with that evidence documented directly in the clip's own warnings, not silently. - A systematic SAM2 behavior: the very last trackable frame before an object leaves the frame or gets caught consistently shows an undersized mask (confirmed independently across 3 of the 4 clips — each one's apparent object size collapsed abruptly on its final frame only). Excluding that one artifact frame turned physically-impossible speed spikes (as high as 104 m/s for a hand toss) into the clean, textbook parabolic curves shown above. A real, generalizable finding worth an automated check in a future pass (see Planned improvements).
Two independent, one-time calibration steps — a camera property and an object property, unrelated to each other:
- Camera intrinsics — a printed checkerboard, recorded in the same
video mode/zoom as the real capture (phones crop the sensor differently
between photo and video modes, so a photo's EXIF focal length doesn't
transfer). Recovers focal length and real lens distortion via
cv2.calibrateCamera. This run: 135/136 sampled frames detected, 0.64px reprojection error. - Object real-world size — a photo of the object next to a coin or
credit card (or, for something too small/large for that to be practical,
a direct measurement/published spec). Full step-by-step protocol,
including why each step matters and answers to questions that came up
in practice, is in
docs/capture-protocol.md.
Skipping camera calibration falls back to an "estimated" tier (a generic assumed field of view) — usable, but honestly labeled as lower accuracy. That penalty is asymmetric: lateral position is mathematically unaffected by a wrong focal-length guess whenever the object's size is known (proven by direct derivation against the actual code, not just claimed); only the depth (toward/away-from-camera) axis and lens-distortion correction are actually at stake.
pip install -r requirements.txt
# One-time: calibrate your camera (record a checkerboard video first)
python -m velocv calibrate --video checkerboard.mov --pattern 9x6 --square-mm 20 --out camera.yaml
# One-time per object: get its real size from a reference photo
python -m velocv size-ref --object-px 300 --reference-px 428 --reference card
# Run
python -m velocv run --video clip.mov --select --frame 0 --fps 30 \
--camera camera.yaml --dim diameter --size 0.0762 --mode auto --out results/Prefer a browser UI over the CLI? python app.py launches a local Gradio
web demo (velocv/ui/webapp.py) at http://127.0.0.1:7860, running at the
estimated-intrinsics tier for casual, drag-and-drop use with no calibration
step required.
- Rolling shutter on phone sensors is mitigated by high-fps capture, not eliminated.
- Flight-mode's gravity-axis fit assumes a level camera by default (a gravity-direction fit relaxes this automatically given enough frames).
- Camera and capture video must share the same zoom/lens setting as the calibration recording.
- Aerodynamic objects (frisbees, badminton shuttles) deviate from the pure ballistic model; large fit residuals trigger an automatic fallback to general mode with an explicit notice, not a silently wrong number.
- Mass estimation is intentionally excluded (see How it works).
- CPU-only: SAM2 has no real-time deadline in this offline pipeline, but it's not fast — a few minutes for a several-second clip on a laptop CPU.
Concrete, prioritized ideas for a future pass, not aspirational hand-waving:
- Measure the reference object with calipers directly rather than a stated/assumed size — the single biggest lever, since size error propagates ~linearly into every downstream number.
- Lock phone focus/exposure before recording to prevent the autofocus hunting documented above.
- Automate detection of the last-frame SAM2 mask-undersizing artifact in the pipeline itself, instead of manual post-hoc trimming.
- Tune the flight-segment search's window/stride/min-length thresholds for short real-world clips — only 1 of 4 clips in this validation run reached the more-accurate flight-mode fit; the other three defaulted to the noisier general-mode fallback partly on frame-count grounds.
- Average multiple independent tosses for a statistically tighter headline accuracy number (the flight-mode result above is a single clip).
Python 3.11+ · PyTorch + SAM2 (Apache-2.0) · Ultralytics YOLO (category classification) · OpenCV · NumPy/SciPy · Matplotlib · Gradio
MIT — see LICENSE.