Skip to content

Taxonomy Comparisons

Jump to bottom
priamai edited this page Mar 27, 2026 · 2 revisions

Opinions on World Models Classifications and Taxonomization

When attempting to categorize the rapidly evolving field of World Models and Physical AI, I have found two prominent frameworks online:

  1. Taxonomy-by-zhuokaiz: A landscape mapped by industry bets and computational paradigms.
  2. Taxonomy-Chris-Paxton: A landscape mapped by mathematical formulations of dynamics and control.

While both attempt to map the same territory (teaching AI to understand and interact with the physical world), they do so through completely different lenses.

1. The "Zhuokai Zhao" Perspective: The Industry & Philosophical Lens

Zhuokai Zhao groups approaches based on macro-level architectural philosophies and the central "bets" companies are making regarding the nature of intelligence.

Key Groupings:

  • JEPA (AMI Labs): The bet on abstract latent-space prediction (avoiding pixel reconstruction).
  • Spatial Intelligence (World Labs): The bet that true understanding requires explicit 3D geometry and persistence.
  • Learned Simulation (Google / Runway): The bet on generative video converging with latent-space RL (e.g., Dreamer, Genie).
  • Physical AI Infrastructure (NVIDIA Cosmos): The structural bet on providing the foundational tooling and tokenizers.
  • Active Inference (VERSES AI): The neurocomputational bet on minimizing surprise via Free Energy (Friston).

This taxonomy answers the question: "Who is building what, and what core assumption are they betting their billions on?"

2. The "Chris Paxton" Perspective: The Mathematical & Control Lens

Chris Paxton approaches the problem from a robotics and control theory standpoint, categorizing models strictly by how they formally construct the "dynamics model" (how the world evolves) and extract the "action" (what the robot does).

Key Groupings:

  • Action-Conditioned ($x' = f(x, a)$): Predicts future state directly from current state and an explicit action vector (e.g., V-JEPA 2, Dreamer v4). Theoretically pure but suffers from compounding errors.
  • Video World Models ($x' = f(x)$ and $a = g(x, x')$): Pure video generators that require an "inverse dynamics" model to work backward and guess the required control actions (e.g., DreamGen). Highly scalable on internet data.
  • Joint World Action Models ($x', a = f(x)$): Implicitly learns both how the scene evolves and what the robot should do simultaneously (e.g., DreamZero).

This taxonomy answers the question: "How does the neural network connect observations to robotic movement?"

Direct Comparison

Feature/Concept Zhuokai Zhao's Taxonomy Chris Paxton's Taxonomy
Primary Organizing Principle Organizational Philosophy & Architecture Types Mathematical Formulation of State ($x$) & Action ($a$)
Number of Primary Bins 5 Categories 3 Formulations
View of JEPA (LeCun) One of five major industry pillars (The "Abstract Prediction" bet). A sub-type of the "Action-Conditioned" category (The purest but hardest formulation).
View of Dreamer (DeepMind) Belongs in "Learned Simulation", alongside generative video. Belongs in "Action-Conditioned", sitting right next to JEPA.
Treatment of 3D Models A fundamental pillar ("Spatial Intelligence") separate from predictive models. Treated as a secondary representation choice ("2D vs 3D") orthogonal to the action logic.
Focus on Data Focuses on architectural capacity (pixels vs embeddings vs 3D). Focuses on the "action label bottleneck" (Internet video lacks action vectors).

Synthesis: Where They Intersect

The two taxonomies are not mutually exclusive; rather, Zhuokai Zhao's categories often map onto the mathematical formulations described by Paxton.

For instance:

  • Zhuokai Zhao’s "JEPA" and the RL-branch of "Learned Simulation" (Dreamer) both fall firmly under Paxton's "Action-Conditioned Models". Even though Zhuokai Zhao separates them based on company lineage and downstream use (pure prediction vs RL policy), Paxton unites them because both explicitly condition their rollout on an action vector $f(x,a)$.
  • Zhuokai Zhao’s "Learned Simulation (Video-generation lineage)" closely aligns with Paxton's "Video World Models with Inverse Dynamics." The video models generate the visual future, and the downstream robotics tasks require inverse dynamics to extract the control logic.
  • Both acknowledge a "new wave": What Paxton calls Joint World Action Models (WAMs) represents the bleeding edge of models trying to fuse deep generation with action prediction. This is a space NVIDIA Cosmos (from Zhuokai Zhao's list) is aggressively targeting with its new reasoning and transfer microservices.

Ultimately, navigating Physical AI requires both lenses: you need Zhuokai Zhao's map to understand where the capital and computational philosophies are flowing, and Paxton's map to understand how the equations are actually wired to the robot's motors.

Clone this wiki locally