Progress and limitations of deep networks to recognize objects in unusual poses

by Amro Abbas & Stephane Deny
ArXiv Link: https://arxiv.org/abs/2207.08034

ObjectPose Dataset

ُThe Datasets folder contains the code for creating ObjectPose datasets. The code for rendering 3d objects in this repo is from Strike (With) A Pose repo.

You can also directly access the dataset of images, 3D models and background images at this address: https://huggingface.co/datasets/Amro-Kamal/ObjectPose/tree/main

Some Samples from the data:

In addition to the main ObjectPose dataset, the paper introduced a set of similar datasets to test to robustness to different tansformations including scaling, rotation, background rotation, etc. See the paper for more details about the datasets.

Deep Networks

How to run the models on ImageNetV2 dataset:

1. Create (only once) and activate python environment (conda create -n "env name") then (conda activate "env name"). Alternatively, if you are not using conda, you can use venv to create the env link.
2. Clone the repo (git clone https://github.com/amro-kamal/ObjectPose.git). This will clone a folder "ObjectPose".
3. cd to ObjectPose directory.
4. Install the requirements (pip install -r requirements.txt).
5. Clone CLIP repo to the ObjectPose/models folder:

cd ObjectPose/src/models                  #go to the models folder 
git clone https://github.com/openai/CLIP  #clone CLIP repo
cd ..                                     #go back to the src folder

5. Download ImageNetV2 dataset from here (1.2GB). Create a folder (call it data for example) and add the dataset folder there (Remenber this dataset path because we will pass it to the code later).
6. Make sure you are at the ObjectPose/src folder (cd src). Then run one of the commands in the commands.txt files based on the dataset you want to use. For python run_imagenet_v2.py for ImageNetv2 dataset. This command explained under the "Testing on ImageNetV2" section below.

Install the requirements:

pip install -r requirements.txt

The command will install all the requirements for you. See the requirements.txt for all the rquirements. The code works fine with Python 3.7.13.

Testing on ImageNetV2

To test the models on ImageNetV2 dataset, go to src folder and run the command:

python run_imagenet_v2.py --batchsize=16
                          --dataroot='../../data/imagenetv2-matched-frequency/data' 
                          --allresultpath = '../../data/imagenetv2-matched-frequency/all_results'
                          --datasetname='imagenetv2' 
                          --modelslist = 'all'
                          --testcode
                          --crop
                          

--dataroot => Dataset path. Change it to your custom path.

--allresultpath => Path to save the result. The code will save a .txt file containing the top1 and top5 accuracy. Change it to your custom path.

--datasetname='imagenetv2' for ImageNetV2 | 'co3d' for CO3D.

--testcode => When this flag is set to True the code will run for one batch only, just to test the code. After you test your code make sure to set it to False.

--modelslist => If 'all', all the models in the model.py file will be tested (currently 37 models). If you want to test specific models, pass a string containing the models' names (space sperated), for example --modelslist = 'ViT_21k_L16 ViT_21k_B16'. See the model.py script for the all the available models.

Testing on ObjectPose:

To test the models on ObjectPose dataset, go the src folder and run the command:

python run_objectpose.py --batchsize=16 --dataroot="../../data/ObjectPose_dataset/ObjectPose" --saveroot="../../data/ObjectPose_dataset/Experiments Results/ObjectPose" --poses="ROLL YAW PITCH" --modelslist='all' --bgs="bg1 bg2 nobg" --crop

--poses => Rotation axes used for creating the data. See commands.txt file for poses corresponding to each dataset.

--bgs => Background images: this defines which part of the data to use. Note that not all the datasets use three background images. See commands.txt file.