Neural Interactive Keypoint Detection

This is the official pytorch implementation of our ICCV 2023 paper "Neural Interactive Keypoint Detection."

Jie Yang, Ailing Zeng, Feng Li, Shilong Liu, Ruimao Zhang, Lei Zhang

Keywords: 👯 Multi-person 2D pose estimation, 💃 Human-in-the-loop, 🤝Interactive model

❤️ Highlights

• Click-Pose has been supported in our DeepDataSpace platform. See details for How to perform intelligent labeling with DDS here.
• All models for COCO, Human-Art, OCHuman, and CrowdPose are released!
• Work flow: 🤖 Model localizes all keypoints -> 👨 User corrects a few wrong keypoints -> 🤖 Model refines other keypoints

💙 Click-Pose

• 👇 We first propose an interactive keypoint detection task for efficient keypoint annotation.

• 👇 We present the first neural interactive keypoint detection framework, Click-Pose, an end-to-end baseline to annotate multi-person 2D keypoints given an image.

• 👇 Click-Pose is more than 10 times faster than manual annotation. Importantly, it significantly alleviates model bias in out-of-domain annotation (e.g., on Human-Art), reducing the time required by 83% compared to state-of-the-art model annotation (ViTPose) with manual correction.

🚀 Model Zoo

1. Model-Only Results

COCO val2017 set

Model	Backbone	Lr schd	mAP	AP50	AP75	APM	APL	Time (ms)	Model
ED-Pose	ResNet-50	60e	71.7	89.7	78.8	66.2	79.7	51	GitHub, Model
Click-Pose	ResNet-50	40e	73.0	90.4	80.0	68.1	80.5	48	Google Drive

Human-Art val set

Model	Backbone	mAP	APM	APL	Model
ED-Pose	ResNet-50	37.5	7.6	41.1	GitHub, Model
Click-Pose	ResNet-50	40.5	8.3	44.2	Google Drive

OCHuman test set

Model	Backbone	mAP	AP50	AP75	Model
ED-Pose	ResNet-50	31.4	39.5	35.1	GitHub, Model
Click-Pose	ResNet-50	33.9	43.4	37.5	Google Drive

Note that the model is trained on COCO train2017 set and tested on COCO val2017 set, Human-Art val set, and OCHuman test set.

2. Neural Interactive Results

In-domain Annotation (COCO val2017)

Model	Backbone	NoC@85	NoC@90	NoC@95	Model
ViTPose	ViT-Huge	1.46	2.15	2.87	GitHub, Model
Click-Pose	ResNet-50	0.95	1.48	1.97	Google Drive

Out-of-domain Annotation (Human-Art val)

Model	Backbone	NoC@85	NoC@90	NoC@95	Model
ViTPose	ViT-Huge	9.12	9.79	10.13	GitHub, Model
Click-Pose	ResNet-50	4.82	5.81	6.45	Google Drive

🔨 Environment Setup

Installation

We use the ED-Pose as our codebase. We test our models under python=3.7.3,pytorch=1.9.0,cuda=11.1. Other versions might be available as well.

1. Clone this repo

git clone https://github.com/IDEA-Research/Click-Pose.git
cd Click-Pose

2. Install Pytorch and torchvision

Follow the instruction on https://pytorch.org/get-started/locally/.

# an example:
conda install -c pytorch pytorch torchvision

3. Install other needed packages

pip install -r requirements.txt

4. Compiling CUDA operators

cd models/clickpose/ops
python setup.py build install
# unit test (should see all checking is True)
python test.py
cd ../../..

Data Preparation

For COCO data, please download from COCO download. The coco_dir should look like this:

|-- Click-Pose
`-- |-- coco_dir
    `-- |-- annotations
        |   |-- person_keypoints_train2017.json
        |   `-- person_keypoints_val2017.json
        `-- images
            |-- train2017
            |   |-- 000000000009.jpg
            |   |-- 000000000025.jpg
            |   |-- 000000000030.jpg
            |   |-- ... 
            `-- val2017
                |-- 000000000139.jpg
                |-- 000000000285.jpg
                |-- 000000000632.jpg
                |-- ... 

For Human-Art data, please download from Human-Art download, The humanart_dir should look like this:

|-- Click-Pose
`-- |-- humanart_dir
    `-- |-- annotations 
        |   |-- training_humanart.json
        |   |-- validation_humanart.json
        `-- images
            |-- 2D_virtual_human
                |-- ...
            |-- 3D_virtual_human
                |-- ...
            |-- real_human
                |-- ...

For CrowdPose data, please download from CrowdPose download, The crowdpose_dir should look like this:

|-- Click-Pose
`-- |-- crowdpose_dir
    `-- |-- json
        |   |-- crowdpose_train.json
        |   |-- crowdpose_val.json
        |   |-- crowdpose_trainval.json (generated by util/crowdpose_concat_train_val.py)
        |   `-- crowdpose_test.json
        `-- images
            |-- 100000.jpg
            |-- 100001.jpg
            |-- 100002.jpg
            |-- 100003.jpg
            |-- 100004.jpg
            |-- 100005.jpg
            |-- ... 

For OCHuman data, please download from OCHuman download. The ochuman_dir should look like this:

|-- Click-Pose
`-- |-- ochuman_dir
    `-- |-- annotations
        `-- images

🥳 Run

Train on COCO:

Model-Only

export CLICKPOSE_COCO_PATH=/path/to/your/coco_dir
 python -m torch.distributed.launch --nproc_per_node=4 main.py \
    --output_dir "logs/ClickPose_Model-Only" \
    -c config/clickpose.cfg.py \
    --options batch_size=4 epochs=100 lr_drop=80 use_ema=TRUE human_feedback=FLASE feedback_loop_NOC_test=FALSE feedback_inference=FALSE only_correction=FALSE \
    --dataset_file="coco" 

Neural Interactive

export CLICKPOSE_COCO_PATH=/path/to/your/coco_dir
 python -m torch.distributed.launch --nproc_per_node=4 main.py \
    --output_dir "logs/ClickPose_Neural_Interactive" \
    -c config/clickpose.cfg.py \
    --options batch_size=4 epochs=100 lr_drop=80 use_ema=TRUE human_feedback=TRUE feedback_loop_NOC_test=FALSE feedback_inference=FALSE only_correction=FALSE \
    --dataset_file="coco"

Evaluation on COCO:

Model-Only

export CLICKPOSE_COCO_PATH=/path/to/your/coco_dir
 python -m torch.distributed.launch --nproc_per_node=4 main.py \
    --output_dir "logs/ClickPose_Model-Only_eval" \
    -c config/clickpose.cfg.py \
    --options batch_size=4 epochs=100 lr_drop=80 use_ema=TRUE human_feedback=FLASE feedback_loop_NOC_test=FALSE feedback_inference=FALSE only_correction=FALSE \
    --dataset_file="coco" \
    --pretrain_model_path "./models/ClickPose_model_only_R50.pth" \
    --eval

Neural Interactive-NoC metric

export CLICKPOSE_COCO_PATH=/path/to/your/coco_dir
export CLICKPOSE_NoC_Test="TRUE"
export CLICKPOSE_SAVE_PATH = "./NoC_95_coco.json"
export NoC_thr = 0.95
    python -m torch.distributed.launch --nproc_per_node=1 --master_port 3458 main.py \
    --output_dir "logs/ClickPose_Neural_Interactive_eval" \
    -c config/clickpose.cfg.py \
    --options batch_size=1 epochs=100 lr_drop=80 use_ema=TRUE human_feedback=TRUE feedback_loop_NOC_test=TRUE feedback_inference=TRUE only_correction=FALSE num_select=20 \
    --dataset_file="coco" \
    --pretrain_model_path "./models/ClickPose_interactive_R50.pth" \
    --eval

Neural Interactive-AP metric

export CLICKPOSE_COCO_PATH=/path/to/your/coco_dir
export CLICKPOSE_NoC_Test="TRUE"
for CLICKPOSE_Click_Number in {1..17}
do
    python -m torch.distributed.launch --nproc_per_node=4 --master_port 3458 main.py \
    --output_dir "logs/ClickPose_Neural_Interactive_eval" \
    -c config/clickpose.cfg.py \
    --options batch_size=4 epochs=100 lr_drop=80 use_ema=TRUE human_feedback=TRUE feedback_loop_NOC_test=FALSE feedback_inference=TRUE only_correction=FALSE num_select=20 \
    --dataset_file="coco" \
    --pretrain_model_path "./models/ClickPose_interactive_R50.pth" \
    --eval
done

Evaluation on Human-Art:

Model-Only

export CLICKPOSE_HumanArt_PATH=/path/to/your/humanart_dir
 python -m torch.distributed.launch --nproc_per_node=4 main.py \
    --output_dir "logs/ClickPose_Model-Only_eval" \
    -c config/clickpose.cfg.py \
    --options batch_size=4 epochs=100 lr_drop=80 use_ema=TRUE human_feedback=FLASE feedback_loop_NOC_test=FALSE feedback_inference=FALSE only_correction=FALSE \
    --dataset_file="humanart" \
    --pretrain_model_path "./models/ClickPose_model_only_R50.pth" \
    --eval

Neural Interactive-NoC metric

export CLICKPOSE_HumanArt_PATH=/path/to/your/humanart_dir
export CLICKPOSE_NoC_Test="TRUE"
export CLICKPOSE_SAVE_PATH = "./NoC_95_humanart.json"
export NoC_thr = 0.95
    python -m torch.distributed.launch --nproc_per_node=1 --master_port 3458 main.py \
    --output_dir "logs/ClickPose_Neural_Interactive_eval" \
    -c config/clickpose.cfg.py \
    --options batch_size=1 epochs=100 lr_drop=80 use_ema=TRUE human_feedback=TRUE feedback_loop_NOC_test=TRUE feedback_inference=TRUE only_correction=FALSE num_select=20 \
    --dataset_file="humanart" \
    --pretrain_model_path "./models/ClickPose_interactive_R50.pth" \
    --eval

Neural Interactive-AP metric

export CLICKPOSE_HumanArt_PATH=/path/to/your/humanart_dir
export CLICKPOSE_NoC_Test="TRUE"
for CLICKPOSE_Click_Number in {1..17}
do
    python -m torch.distributed.launch --nproc_per_node=4 --master_port 3458 main.py \
    --output_dir "logs/ClickPose_Neural_Interactive_eval" \
    -c config/clickpose.cfg.py \
    --options batch_size=4 epochs=100 lr_drop=80 use_ema=TRUE human_feedback=TRUE feedback_loop_NOC_test=FALSE feedback_inference=TRUE only_correction=FALSE num_select=20 \
    --dataset_file="humanart" \
    --pretrain_model_path "./models/ClickPose_interactive_R50.pth" \
    --eval
done

Evaluation on OCHuman:

Model-Only

export CLICKPOSE_OCHuman_PATH=/path/to/your/ochuman_dir
 python -m torch.distributed.launch --nproc_per_node=4 main.py \
    --output_dir "logs/ClickPose_Model-Only_eval" \
    -c config/clickpose.cfg.py \
    --options batch_size=4 epochs=100 lr_drop=80 use_ema=TRUE human_feedback=FLASE feedback_loop_NOC_test=FALSE feedback_inference=FALSE only_correction=FALSE \
    --dataset_file="ochuman" \
    --pretrain_model_path "./models/ClickPose_model_only_R50.pth" \
    --eval

Neural Interactive-NoC metric

export CLICKPOSE_OCHuman_PATH=/path/to/your/ochuman_dir
export CLICKPOSE_NoC_Test = "TRUE"
export CLICKPOSE_SAVE_PATH = "./NoC_95_ochuman.json"
export NoC_thr = 0.95
    python -m torch.distributed.launch --nproc_per_node=1 --master_port 3458 main.py \
    --output_dir "logs/ClickPose_Neural_Interactive_eval" \
    -c config/clickpose.cfg.py \
    --options batch_size=1 epochs=100 lr_drop=80 use_ema=TRUE human_feedback=TRUE feedback_loop_NOC_test=TRUE feedback_inference=TRUE only_correction=FALSE num_select=20 \
    --dataset_file="ochuman" \
    --pretrain_model_path "./models/ClickPose_interactive_R50.pth" \
    --eval

Neural Interactive-AP metric

export CLICKPOSE_OCHuman_PATH=/path/to/your/ochuman_dir
export CLICKPOSE_NoC_Test="TRUE"
for CLICKPOSE_Click_Number in {1..17}
do
    python -m torch.distributed.launch --nproc_per_node=4 --master_port 3458 main.py \
    --output_dir "logs/ClickPose_Neural_Interactive_eval" \
    -c config/clickpose.cfg.py \
    --options batch_size=4 epochs=100 lr_drop=80 use_ema=TRUE human_feedback=TRUE feedback_loop_NOC_test=FALSE feedback_inference=TRUE only_correction=FALSE num_select=20 \
    --dataset_file="ochuman" \
    --pretrain_model_path "./models/ClickPose_interactive_R50.pth" \
    --eval
done

Cite Click-Pose

If you find this repository useful for your work, please consider citing it as follows:

@inproceedings{yang2023neural,
  title={Neural Interactive Keypoint Detection},
  author={Yang, Jie and Zeng, Ailing and Li, Feng and Liu, Shilong and Zhang, Ruimao and Zhang, Lei},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={15122--15132},
  year={2023}
}

@inproceedings{yang2022explicit,
  title={Explicit Box Detection Unifies End-to-End Multi-Person Pose Estimation},
  author={Yang, Jie and Zeng, Ailing and Liu, Shilong and Li, Feng and Zhang, Ruimao and Zhang, Lei},
  booktitle={The Eleventh International Conference on Learning Representations},
  year={2022}
}