HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objects

This repository contains the content of the following paper:

HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objects
Xintao Lv ^1,* , Liang Xu ^1,2* , Yichao Yan ¹, Xin Jin ², Congsheng Xu ¹, Shuwen Wu¹, Yifan Liu¹, Lincheng Li³, Mengxiao Bi³, Wenjun Zeng², Xiaokang Yang ^1
1 Shanghai Jiao Tong University , ² Eastern Institute of Technology, Ningbo, ³ NetEase Fuxi AI Lab

Dataset Download

Please fill out this form to request authorization to download HIMO for research purposes. After downloading the dataset, unzip the data in ./data and you'll get the following structure:

./data
|-- joints
|   |-- S01T001.npy
|   |-- ...
|-- smplx
|   |-- S01T001.npz
|   |-- ...
|-- object_pose
|   |-- S01T001.npy
|   |-- ...
|-- text
|   |-- S01T001.txt
|   |-- ...
|-- segments
|   |-- S01T001.json
|   |-- ...
|-- object_mesh
|   |-- Apple.obj
|   |-- ...

Data Visualization

We use the AIT-Viewer to visualize the dataset. You can follow the instructions below to visualize it.

pip install -r visualize/requirements.txt

You also need to download the SMPL-X models and place them in ./body_models, which should look like:

./body_models
|-- smplx
    ├── SMPLX_FEMALE.npz
    ├── SMPLX_FEMALE.pkl
    ├── SMPLX_MALE.npz
    ├── SMPLX_MALE.pkl
    ├── SMPLX_NEUTRAL.npz
    ├── SMPLX_NEUTRAL.pkl
    └── SMPLX_NEUTRAL_2020.npz

Then you can run the following command to visualize the dataset.

# Visualize the skeleton
python visualize/skel_viewer.py
# Visualize the SMPLX
python visualize/smplx_viewer.py
# Visualize the segment data
python visualize/segment_viewer.py

Training

To train the model in 2-object setting, run

python -m src.train.train_net_2o --exp_name net_2o --num_epochs 1000

To train the model in 3-object setting, run

python -m src.train.train_net_3o --exp_name net_3o --num_epochs 1000

To evaluate the model, you need to train your own evaluator or use the checkpoint we provide here (put them under ./save). Then run

python -m src.eval.eval_himo_2o

or

python -m src.eval.eval_himo_3o

Visualization