The Jump Benchmark

A Physics-Aware Video Generation Benchmark for 3D World Models

Testing whether video generation models can reason about real-world physics, human intentionality, and chaotic kinetic events — not just predict pixels.

---

Overview

The Jump Benchmark is a demanding evaluation suite for video generation models centered on a single, information-dense real-world event: a person climbing onto the roof of a moving Sprinter van, performing to a crowd, then intentionally jumping off at a musical beat drop — resulting in catastrophic impact with the ground.

This ~15-second sequence encodes an extraordinary density of physics phenomena, emotional dynamics, and temporal structure that current video generation models (Sora, Veo, Kling, Open-Sora, etc.) cannot faithfully reproduce. The benchmark is designed to expose fundamental weaknesses in 2D pixel-prediction approaches and push the field toward true 3D physics-aware world models.

Why This Sequence?

This event is a uniquely powerful test case because it compresses into seconds:

Dimension	What It Tests
Multi-phase dynamics	Climb → performance → leap → freefall → impact → collapse
Intentionality	The jump is deliberate and timed to music — not accidental
Discontinuous physics	Smooth motion → sudden acceleration → violent deceleration
Audio-visual coupling	Beat drop triggers the physical event
Crowd dynamics	Collective reaction shifts from excitement to horror
Emotional gradient	Euphoria → commitment → terror → agony
3D spatial reasoning	Van roof height, arc trajectory, ground plane, body orientation

No existing benchmark captures this combination.

---

Benchmark Structure

Difficulty Tiers

Tier	Name	Description
T1	Exact Reconstruction	Reproduce the original sequence frame-by-frame from sparse keyframes
T2	Temporal Interpolation	Fill in missing frames at 2x, 4x, 8x temporal gaps
T3	Slow-Motion Synthesis	Generate 240fps slow-motion of the impact sequence
T4	Novel Viewpoint	Synthesize the event from rotated/elevated camera angles
T5	Physics Perturbation	Altered gravity (0.5g, 2g, lunar), altered mass, altered surface
T6	Counterfactual	"What if he landed on a trampoline?" / "What if he tucked and rolled?"
T7	Extreme Stress	Reverse time, infinite loop at impact, 1000fps micro-analysis

Core Metrics

Metric	Abbreviation	What It Measures
Gradient Coherence	GC	Alignment of motion/energy vectors across consecutive frames
Phase Interpolation Score	PIS	Smooth transitions between distinct physical phases
3D Vector Fidelity	3VF	Accuracy of inferred 3D trajectories vs. ground truth
Impulse Realism	IR	Whether impact forces match real biomechanical models
Temporal Warping Error	TWE	Drift from ground-truth timing across the sequence
Beat Sync Accuracy	BSA	Alignment of physical events with audio beat markers
Intentionality Coherence	IC	Whether the model preserves deliberate action semantics
Crowd Reaction Fidelity	CRF	Temporal accuracy of collective emotional response

Composite Score

JumpScore = 0.20*GC + 0.15*PIS + 0.20*3VF + 0.15*IR + 0.10*TWE + 0.08*BSA + 0.07*IC + 0.05*CRF

A model must achieve JumpScore >= 0.85 across all T1-T3 tiers to be considered "Jump-Complete."

---

Quick Start

pip install -e .

from jumpbench import JumpBenchmark, evaluate

benchmark = JumpBenchmark(tiers=["T1", "T2", "T3"])
results = evaluate(
    model_outputs="path/to/generated_videos/",
    ground_truth="path/to/ground_truth/",
    config="configs/default.yaml"
)
results.summary()
results.export_leaderboard("results/leaderboard.json")

---

Repository Structure

jumpbenchmark/
├── README.md                    # This file
├── setup.py                     # Package installation
├── configs/
│   └── default.yaml             # Default evaluation config
├── jumpbench/
│   ├── __init__.py              # Package init + public API
│   ├── benchmark.py             # Core benchmark orchestrator
│   ├── metrics/
│   │   ├── __init__.py
│   │   ├── gradient_coherence.py
│   │   ├── phase_interpolation.py
│   │   ├── vector_fidelity.py
│   │   ├── impulse_realism.py
│   │   ├── temporal_warping.py
│   │   └── composite.py
│   ├── data/
│   │   ├── __init__.py
│   │   └── dataset.py
│   ├── prompts/
│   │   ├── __init__.py
│   │   └── templates.py
│   └── utils/
│       ├── __init__.py
│       ├── physics.py
│       └── video_io.py
├── architectures/
│   └── proposals.md
├── demo/
│   └── concept.md
└── docs/
    └── specification.md

---

Leaderboard

Rank	Model	JumpScore	GC	PIS	3VF	IR	TWE	Tier
—	Baseline (copy last frame)	0.12	0.08	0.05	0.10	0.15	0.22	T1
—	Your model here	—	—	—	—	—	—	—

---

Why This Matters

See docs/specification.md for the full argument on why the Jump Benchmark can become the definitive test for physics-aware video generation — the way GLUE defined NLU and VBench defined video quality.

Architectural Innovations

See architectures/proposals.md for novel model architecture proposals designed to excel on this benchmark.

Demo Concept

See demo/concept.md for the viral demonstration strategy.

---

Citation

@misc{jumpbenchmark2026,
  title={The Jump Benchmark: A Physics-Aware Evaluation Suite for Video World Models},
  year={2026},
  url={https://github.com/direncode/jumpbenchmark}
}

License

Apache 2.0