IKOL

Code repository for the paper:

IKOL: Inverse kinematics optimization layer for 3D human pose and shape estimation via Gauss-Newton differentiation

Juze Zhang, Ye Shi, Yuexin Ma, Lan Xu, Jingyi Yu, Jingya Wang

[Paper] [Project Page]

In AAAI 2023

Installation instructions

# 1. Create a conda virtual environment.
conda create -n IKOL python=3.8 pytorch=1.11 cudatoolkit=11.3 torchvision -c pytorch -y
conda activate IKOL

# 2. Install mmcv-full
pip install openmim
mim install mmcv-full

# 3. Install PyTorch3D
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install -c bottler nvidiacub -y
conda install pytorch3d -c pytorch3d -y

# 4. Pull our code
git clone https://github.com/Juzezhang/IKOL.git
cd IKOL

# 5. Install
pip install -e .

Pretrained models and Dataset

• Download the SMPL model basicModel_neutral_lbs_10_207_0_v1.0.0.pkl from here at common/utils/smplpytorch/smplpytorch/native/models.
• Download our pretrained model [ Google Drive] to re-implement our results in Tab. 1 of the main paper.
• Thanks to the great job done by Li et al, we use the dataset formats provided in HybrIK

Evaluation

python ./scripts/validate.py --batch 64 --gpus 0 --world-size 1 --launcher pytorch --rank 0 --dist-url tcp://${HOST}:${PORT} --cfg ${CONFIG} --checkpoint ${CKPT} --flip-test --dataset 3DPW/3DHP --loader_type mmloader/hybrik_loader

e.g. evaluate on 3DPW

CUDA_VISIBLE_DEVICES=0 python ./scripts/validate.py --batch 64 --gpus 0 --world-size 1 --launcher pytorch --rank 0 --dist-url tcp://127.0.0.1:23456 --cfg ./configs/IKOL_resnet34_mmloader.yaml --checkpoint ./checkpoints/pretrained_3dpw_finetune_hybrikloader.pth --flip-test --dataset 3DPW --loader_type hybrikloader

Training

Our approach incorporates two distinct data loader formats, sourced from the HybrIK repository and mmhuman3d. For the fine-tuning process, we have utilized the HybrIK loader. In addition, we train our own model from the stretch with mmloader. Here two implement example. Below are examples of both implementations.

Training with hybrik loader(only finetune):

python ./scripts/train_hybrikloader.py -nThreads 0 --launcher pytorch --rank 0 --dist-url tcp://127.0.0.1:23456 --exp-id train_res34 --cfg ./configs/IKOL_resnet34_hybrikloader_finetune.yaml --seed 123123

Training with mmloader:

python ./scripts/train_mmloader.py -nThreads 0 --launcher pytorch --rank 0 --dist-url tcp://127.0.0.1:23456 --exp-id train_res34 --cfg ./configs/IKOL_resnet34_mmloader.yaml --seed 123123

Acknowledgement

The implementation was heavily built upon HybrIK and mmhuman3d. We thank the authors for their generosity to release code.

Citing

If our code helps your research, please consider citing the following paper:

  @inproceedings{
        zhang2023ikol,
        title={IKOL: Inverse kinematics optimization layer for 3D human pose and shape estimation via Gauss-Newton differentiation},
        author={Juze Zhang and Ye Shi and Yuexin Ma and Lan Xu and Jingyi Yu and Jingya Wang},
        booktitle={Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)},
        year={2023},
  }