Towards Human-Level 3D Relative Pose Estimation: Generalizable, Training-Free, with Single Reference

Official PyTorch Implementation of Towards Human-Level 3D Relative Pose Estimation: Generalizable, Training-Free, with Single Reference. The codes are primarily contributed by Yajing Luo, Junhong Wang, and Yuan Gao.

Please also refer to our video for more information.

If this code is helpful to your research, please consider citing our paper by:

@article{towards2024,
    title={Towards Human-level 3D Relative Pose Estimation: Zero-Shot Unseen Generalization, Label/Training-Free, and A Single Reference},
    author={Yuan Gao and Yajing Luo and Junhong Wang and Kui Jia and Gui-Song Xia},
    year={2024},
    journal={arXiv preprint arXiv:2406.18453}
}

Setup

1. Python Environment and Dependencies

conda create -n relativ_pose_estimation python=3.9.16
pip install -r requirements.txt

2. Compile nvidiffrast_culling

cd nvdiffrast_culling
pip install .

3. Download Pretrained DINOv2 Model

mkdir third_party & cd third_party
git clone https://github.com/facebookresearch/dinov2.git
cd dinov2 & mkdir weights & cd weights
wget https://dl.fbaipublicfiles.com/dinov2/dinov2_vitl14/dinov2_vitl14_pretrain.pth

4. Download Dataset

mkdir bop_datasets & cd bop_datasets
wget https://huggingface.co/datasets/bop-benchmark/datasets/resolve/main/lm/lm_test_all.zip # LineMOD real image datasets
unzip lm_test_all.zip

5. Query-Reference Pairs Generation

The generated query-reference pairs for the LineMOD dataset that used in our experiments are uploaded at data/gd_pairs. Otherwise, the query-reference pairs can be automatically generated by those codes if data/gd_pairs is empty.

Evaluation (Note That Our Method Is Label/Training-Free)

Evaluate our method on the LineMOD dataset by running:

python run_lm.py

Acknowledgements:

1. Our nvdiffrast_culling is modified based on nvdiffrast.
2. The semantic features are extracted by DINO v2.