SceneMDM: Harnessing Scene Interactions with Affordances for Text-to-Motion

The official PyTorch implementation of the paper "Harnessing Scene Interactions with Affordances for Text-to-Motion".

Getting started

This code was tested on Ubuntu 18.04.5 LTS and requires:

• Python 3.9
• conda3 or miniconda3
• CUDA capable GPU (one is enough)

1. Setup environment

Install ffmpeg (if not already installed):

sudo apt update
sudo apt install ffmpeg

For windows use this instead.

Setup conda env:

conda env create -f environment.yml
conda activate PriorMDM
python -m spacy download en_core_web_sm
pip install git+https://github.com/openai/CLIP.git
pip install git+https://github.com/GuyTevet/smplx.git

2. Get MDM dependencies

PriorMDM share most of its dependencies with the original MDM. If you already have an installed MDM from the official repo, you can save time and link the dependencies instead of getting them from scratch.

If you already have an installed MDM

Link from installed MDM

Before running the following bash script, first change the path to the full path to your installed MDM

bash prepare/link_mdm.sh

First time user

Download dependencies:

bash prepare/download_smpl_files.sh
bash prepare/download_glove.sh
bash prepare/download_t2m_evaluators.sh

Get HumanML3D dataset (For all applications):

Follow the instructions in HumanML3D, then copy the result dataset to our repository:

cp -r ../HumanML3D/HumanML3D ./dataset/HumanML3D

4. Download the pretrained models

Comming soon.

Motion Synthesis

Acknowledgments

This code is standing on the shoulders of giants. We want to thank the following contributors that our code is based on:

PriorMDM, MDM, POSA, guided-diffusion, MotionCLIP, text-to-motion, actor, joints2smpl, TEACH.

License

This code is distributed under an MIT LICENSE.

Note that our code depends on other libraries, including CLIP, SMPL, SMPL-X, PyTorch3D, and uses datasets that each have their own respective licenses that must also be followed.