Skip to content


Repository files navigation

Structured Object-Aware Physics Prediction for Video Modelling and Planning

See or for further information.

Example Animations

Video Prediction

Real Ours VRNN SQAIR DDPAE Linear Supervised
Real Ours VRNN SQAIR DDPAE Linear Supervised

The above depicts the reconstruction and prediction errors of the various models. The models are given 8 frames of video as input, which they reconstruct. Conditioned on this input, all models predict the following 92 frames. Only STOVE manages to generate visually convincing physically behavior over longer timeframes. 10 different sequences of length 100 are shown.

Model-Based Control

MCTS on STOVE MCTS on Real Env. PPO on Env. States PPO on Env. Images

The above shows the performance of the compared models in the interactive environments. The agent controls the red ball and negative reward is given whenever the red ball collides with any other ball. STOVE is used as a world model, predicting future states, frames, and rewards. MCTS can then be used on STOVE for model-based control. We compare to MCTS on the real environment states, as well as PPO on the environment states and raw images. Again, 10 different sequences of length 100 are shown.


Humans can easily predict future outcomes even in settings with complicated interactions between objects. For computers, however, learning models of interactions from videos in an unsupervised fashion is hard. In this paper, we demonstrate that structure and compositionality are key to solving this problem. Specifically, we develop a novel video prediction model from two distinct components: a scene model and a physics model. Both models are compositional and exploit repeating elements wherever possible. We impose a highly structured bottleneck between model components to allow for fast learning and clearly interpretable functionality, without losing any generality or performance. This fully compositional approach yields a strong video prediction model, which clearly outperforms relevant baselines. We produce realistic looking physical behaviour over a possibly infinite time frame and perform competitively even compared to a supervised approach. Finally, we demonstrate the strength of our model as a simulator for sample efficient model-based reinforcement learning in tasks with heavily interacting objects.

Other animations

STOVE's rollouts are stable for a possibly infinite number of timesteps. (Shown are 2000 frames of rollout and we tested up to 100000.)

All components of STOVE scale well to videos with larger number of objects!


Please cite our work here and at as

  title={Structured Object-Aware Physics Prediction for Video Modeling and Planning},
  author={Kossen, Jannik and Stelzner, Karl and Hussing, Marcel and Voelcker, Claas and Kersting, Kristian},
  booktitle={Proceedings of the International Conference on Learning Representations},


Run --create-data to generate billiards and gravity data. Also random data collected in the RL setting to train the action- conditioned world model is generated.


Run bash run_files/ to train the model on billiards and gravity data. Also an actioned-conditioned world model is trained on the avoidance task.


python --interactive allows you to either play in a live environment or a model simulation of the environment.


If you have any questions or problems regarding the code or paper do not hesitate to contact us.


Structured Object-Aware Physics Prediction for Video Modeling and Planning







No releases published


No packages published