| From headset + controllers (reliable) | From headset + hand-tracking (unreliable) |
|---|---|
 |
 |
In extended reality (XR), generating full-body motion of the users is important to understand their actions, drive their virtual avatars for social interaction, and convey a realistic sense of presence. While prior works focused on spatially sparse and always-on input signals from motion controllers, many XR applications opt for vision-based hand tracking for reduced user friction and better immersion. Compared to controllers, hand tracking signals are less accurate and can even be missing for an extended period of time. To handle such unreliable inputs, we present Rolling Prediction Model (RPM), an online and real-time approach that generates smooth full-body motion from temporally and spatially sparse input signals. Our model generates 1) accurate motion that matches the inputs (i.e., tracking mode) and 2) plausible motion when inputs are missing (i.e., synthesis mode). More importantly, RPM generates seamless transitions from tracking to synthesis, and vice versa. To demonstrate the practical importance of handling noisy and missing inputs, we present GORP, the first dataset of realistic sparse inputs from a commercial virtual reality (VR) headset with paired high quality body motion ground truth. GORP provides >14 hours of VR gameplay data from 28 people using motion controllers (spatially sparse) and hand tracking (spatially and temporally sparse). We benchmark RPM against the state of the art on both synthetic data and GORP to highlight how we can bridge the gap for real-world applications with a realistic dataset and by handling unreliable input signals. Our code, pretrained models, and GORP dataset will be released upon acceptance.
[2025-04-07] Our paper is available in Arxiv.
This code was tested on Python 3.10 + PyTorch 2.5.1. While other versions might work as well, we recommend using our conda environment to avoid unexpected issues.
-
Download the human_body_prior lib and body_visualizer lib and put them in this repo.
-
Register + download the SMPL-X model from this link. Download + unzip the 'SMPL-X with removed head bun (NPZ)'. Then move the
SMPLX_NEUTRAL.npzfile to this repo into:./SMPL/smplx/neutral/model.npz.
The repo should look like
RPM
├── body_visualizer
├──── mesh/
├──── tools/
├──── ...
├── human_body_prior/
├──── body_model/
├──── data/
├──── ...
├── SMPL/
├──── smplx/
├────── neutral/
├──────── model.npz
├── data_loaders/
└── ...
- Setup conda env:
conda env create -f environment.yml
conda activate rpmThis README file contains instructions on how to train/evaluate the model and visualize the results.
- Model weights:
The pre-terained model weights used in the paper are available here
The data is only provided upon request via requestor’s email under the CC BY-NC license.
To request access to the GORP dataset you can fill this request form link
If you find our work helpful or use the GORP dataset, please cite:
@inproceedings{barquero2025rolling,
title={From Sparse Signal to Smooth Motion: Real-Time Motion Generation with Rolling Prediction Models},
author={Barquero, German and Bertsch, Nadine and Marramreddy, Manojkumar and Chacón, Carlos and Arcadu, Filippo and Rigual, Ferran and He, Nicky and Palmero, Cristina and Escalera, Sergio and Ye, Yuting and Kips, Robin},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2025},
}This project has used and is built on source codes shared by body_visualizer, human_body_prior, AvatarPoser, AGRoL. We thank the authors for their effort and open sourcing their works.
The majority of RPM code is licensed under CC-BY-NC, however portions of the project are available under separate license terms:
- Trimesh, AvatarPoser and MDM are licensed under the MIT license;
- Human Body Prior is licensed under a custom license for non-commercial scientific research purposes, available at link;
- Body Visualizer is licensed under a custom license for non-commercial scientific research purposes, available at link.