labelImg3D

RGB Image Object Pose Annotation with Python

Description

LabelImg3D is a labeling tool for image graphics. It is written in Python and uses Qt for its graphical interface. The annotations are saved as json files and used in the SMPOE network.

Purpose

We human beings can do 3D reconstruction from single image. Instead of depth estimation from image, we humans beings seems to use a different way, which is to call 3D model in mind, re-align these 3D models and change hyper-parameters of 3D models in the virtual world of our brain. Therefore, why not try to find a way to realign 3D models by single image instead of directly estimating depth.

In this work, we try to find a way to re-align 3D models in 3D space guided by the single images. There are two steps needs to be done, as follows:

• Image-guided 3D Re-Alignment: object detection, 3D model matching, and 3D model pose estimation.
• Image-guided 3D Hyper-parameter of variants model estimation.

---

Features

• Image annotation of the object's true pose relative to the camera.
• Annotation of the target's 2D boxes and 3D boxes in the image.
• GUI customization (predefined labels, auto-saving, etc).
• Exporting linemod dataset for 6D Pose Estimation.

---

Requirements

• Ubuntu / macOS / Windows
• Python3
• PyQt5
• VTK
• numpy
• pandas
• scipy
• PyYAML
• opencv-python

---

Installation

Install labelImg3D

There are two ways for installing the labeltools:

1. install from the released .exe

or

2. run from the code, by the following sh

conda create -n pylabelimg3D python=3.8
conda activate pylabelimg3D
pip install -r requirements.txt
python labelImg3d.py

Packages

If you want to make the installer, you can use the pyinstaller. Try the following command:

pip install pyinstaller
pyinstaller --clean -y LabelImg3D.spec

---

Usage

Data Preparation

Please download the demo scene and open with LabelImg3D.

Scenes
├── Scene1
│   ├── images
│   │   ├──  000000.png ......
│   ├── models
│   │   ├── Car.obj bus.obj ......
|   ├── annotations
|   │   ├── 000000.json ......
├── Scene2
├── ├── images
│   │   ├── ......

Run LabelImg3D

labelImg3D  # open gui

or

python3 labelImg3D.py

Steps

1. Build and launch using the instructions above.

2. Click 'Load Scenes' in Menu/File, and choose the scene folder as Data Preparation.

3. Click 'System Config' in Menu/Config.

4. Click "camera" at System_config Page, and set the FOV to the field of view of the camera taking the scene images.

5. Click "model" at System_config page, and you can set the initial position, max position, position accuracy, size accuracy, scaling factor of the model.

   • initial position: The initial position of the model loaded into the scene.
   • max position: The position of the model in the scene closest to the camera. Moving the model beyond this position resets the model to its initial position during the move.
   • position accuracy: Display accuracy of model x,y,z position, located in 3DProperty module.
   • size accuracy: Display accuracy of model size(w, l, h), located in 3DProperty module.
   • scaling factor: Model movement speed in Z-axis.

6. Click the model name at Models module, and we can see the model loaded to its initial position.

   • If a model is selected and loaded as described above, if both model categories are the same, the new model will be loaded to the selected model location.

7. You can press the Shift key and press the left mouse button to select the model and move the mouse position to move the model in X and Y axis.

8. You can press the right mouse button to select the model and move the mouse position to move the model in the Z-axis.

9. In the visual perception of the human eye, object labeling is considered to be over when the model in the scene completely overlaps with the image object.

The annotation will be saved to the Scenes//annotations.

You can refer to the below hotkeys to speed up your workflow.

Hotkeys

Hot key	Description
Ctrl + s	Save
Ctrl + o	Load scenes
Ctrl + c	Copy model
Ctrl + v	Paste model
Ctrl + Space	Copy Scene
1	Previous image
Space	Next image
x	Delete Model
Shift + Left Mouse Button Move	Move the model along X, Y axis
Right Mouse Button Move	Move the model along Z axis
w	Rotate around the positive x-axis
s	Rotate around the negative x-axis
e	Rotate around the positive y-axis
q	Rotate around the negative y-axis
a	Rotate around the positive z-axis
d	Rotate around the negative z-axis

Label Annotations

The annotation files contain 3 main sections, as follows:

Annotation.json
├── image_file ¹: "images/0000.png"
├── model ²
│ ├── num ³: 5
│ ├── 0 ⁴
│ │ ├── model_file ⁵: "models/Car.obj"
│ │ ├── matrix: [0, …, ] # (16x1) matrix ⁶
│ │ ├── R_matrix_c2o:[0, …, ] # (9x1) matrix ⁷
│ │ ├── T_matrix_c2o:[0, …, ] # (3x1) matrix ⁸
│ │ ├── 2d_bbox: [700, …, ] # (4x1) matrix ⁹
│ │ ├── 3d_bbox: [[721,500] …, ] # (8x2) matrix ¹⁰
│ │ ├── 3d_bbox_w: [[3.8, 2.4, 1.4],…，] #(3x1) matrix ¹¹
│ │ ├── class: 1 # object class num ¹²
│ │ ├── class_name: Car # object class name ¹³
│ │ ├── size: [1.99, 1.55, 18.24] # (3x1) matrix ¹⁴
│ ├── 1⁴
│ │ ├── model_file ⁵: "models/Car.obj"
│ │ ├── ......
├── camera ¹⁵
│ ├── matrix: [1.0, …, ] # (16x1) matrix ¹⁶
│ ├── position: [0.0, 0.0, 0.52] # (3x1) matrix ¹⁷
│ ├── focalPoint: [0.0, 0.0, 0.0] # (3x1) matrix ¹⁸
│ ├── fov: 88.0 # camera fov ¹⁸
│ ├── viewup: [0.0, 1.0, 0.0] # camera viewup ¹⁸
│ ├── distance: 0.52 # camera distance ¹⁸

1. image_file: The labeled image file path.

2. model: Labeling results of objects

3. model──num: The number of objects

4. 0, 1, 2...: each labeled object

5. model_file: the object model path

6. matrix: the pose of object in twin space

7. R_matrix_c2o: Rotation matrix of the object from its own coordinate to the camera coordinate.

8. T_matrix_c2o: Translation matrix of the object from its own coordinate to the camera coordinate.

9. 2d_bbox: 2D bounding box of the object in the image.

10. 3d_bbox: 3D bounding box of the object in the image.

11. 3d_bbox_w: 3D bounding box of the object in the camera coordinate in twin space.

12. class: the object class number.

13. class_name: the object class name.

14. size: the object size (Unit: meter).

15. camera:

16. matrix: the pose of camera in twin space.

17. position: the position of the camera in twin space.

18. focalPoint, fov, viewup, distance: Camera parameters

---

Dataset

This is a dataset named KITTI-6DoF

Framework

---

Developing

git clone https://github.com/CongliangLi/LabelImg3D.git
cd labelImg3D

---

Third-Party Library

name	desc
pyqtconfig	for gui configure

---

Author Home

Congliang Li
Shijie Sun

License

The methods provided on this page are published under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License . This means that you must attribute the work in the manner specified by the authors, you may not use this work for commercial purposes and if you alter, transform, or build upon this work, you may distribute the resulting work only under the same license. If you are interested in commercial usage you can contact us for further options.

Commercial Cooperation, Contact shijieSun@chd.edu.cn