Panoramic localization library containing PyTorch implementations of various panoramic localization algorithms: PICCOLO (ICCV 2021), CPO (ECCV 2022), LDL (ICCV 2023), and FGPL (CVPR 2024). Please refer to the links below to run each of the localization algorithms.
PICCOLO is a panoramic localization algorithm that estimates camera pose given a set of input query image and colored point cloud.
The algorithm optimizes the discrepancy between the query image colors and the projected point cloud's color values.
For more details, please refer to the original paper and video.
CPO is a follow-up work of PICCOLO that also estimated camera pose given a query image and point cloud.
Instead of directly optimizing the color values as in PICCOLO, CPO aims to match the color distributions, which lead to large amounts of runtime improvement.
For more details, please refer to the original paper and video.
LDL is a line-based panoramic localization algorithm that localized a query image with respect to a 3D line map.
To this end, LDL compares the distribution of lines using line distance functions, which makes pose search even faster than CPO.
For more details, please refer to the original paper and video.
FGPL is a line-based localization algorithm that finds the camera pose solely using the geometry of lines and their intersections.
The algorithm first performs coarse pose search using the spatial distribution of lines and their intersections, and then refines the poses by aligning the lines on the sphere.
For more details, please refer to the original paper (to be updated) and video (to be updated).
First ownload the panorama images (pano) and poses (pose) from the following link (download the one without XYZ) and the point cloud (pcd_not_aligned) from the following link.
Also, download the 3D line segments through the following link.
Then, place the data in the directory structure below.
panoramic-localization/data
└── stanford (Stanford2D-3D-S Dataset)
├── pano (panorama images)
│ ├── area_1
│ │ └── *.png
│ ⋮
│ │
│ └── area_6
│ └── *.png
├── pcd (point cloud data)
│ ├── area_1
│ │ └── *.txt
│ ⋮
│ │
│ └── area_6
│ └── *.txt
├── pcd_line (line cloud data)
│ ├── area_1
│ │ └── *.txt
│ ⋮
│ │
│ └── area_6
│ └── *.txt
└── pose (json files containing ground truth camera pose)
├── area_1
│ └── *.json
⋮
│
└── area_6
└── *.json
To obtain results in OmniScenes, please refer to the download instructions below. Note we are using the old version of OmniScenes for this repository. In addition, download the 3D line segments through the following link. Then, place the data in the directory structure below.
panoramic-localization/data
└── omniscenes (OmniScenes Dataset)
├── change_handheld_pano (panorama images)
│ ├── handheld_pyebaekRoom_1_scene_2 (scene folder)
│ │ └── *.jpg
│ ⋮
│ │
│ └── handheld_weddingHall_1_scene_2 (scene folder)
│ └── *.jpg
└── change_handheld_pose (json files containing ground truth camera pose)
| ├── handheld_pyebaekRoom_1_scene_2 (scene folder)
| │ └── *.json
| ⋮
| │
| └── handheld_pyebaekRoom_1_scene_2 (scene folder)
| └── *.json
⋮
└── pcd_line (line cloud data)
| ├── pyebaekRoom_1lines.txt
| │
| ⋮
| │
| └── weddingHall_1lines.txt
└── pcd (point cloud data)
├── pyebaekRoom_1.txt
│
⋮
│
└── weddingHall_1.txt
To run the codebase, you need Anaconda. Once you have Anaconda installed, run the following command to create a conda environment.
conda create --name omniloc python=3.7
conda activate omniloc
pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html
conda install cudatoolkit=10.1
In addition, you must install pytorch_scatter. Follow the instructions provided in the pytorch_scatter github repo. You need to install the version for torch 1.7.0 and CUDA 10.1.
Run the following command for Stanford 2D-3D-S.
python main.py --config config/stanford_piccolo.ini --log log/piccolo_test --method piccolo
Similarly, run the following command for OmniScenes
python main.py --config config/omniscenes_piccolo.ini --log log/piccolo_test --method piccolo
Note the sampling loss optimization can be done in parallel (with a slight sacrifice in localization accuracy), using the following command.
python main.py --config config/omniscenes_piccolo.ini --log log/piccolo_test --method piccolo --override 'refine_parallel=True'
We also provide scripts for directly testing on your own data. First, prepare a query panorama image and 3D colored point cloud. Then, run the following command, and the renderings at the localized pose will be saved in the log directory.
python main.py --config config/omniscenes_piccolo.ini --log LOG_DIRECTORY --method piccolo --single --query_img PATH_TO_QUERY_IMG --color_pcd PATH_TO_COLORED_POINT_CLOUD
Run the following command for Stanford 2D-3D-S.
python main.py --config config/stanford_cpo.ini --log log/cpo_test --method cpo
Similarly, run the following command for OmniScenes
python main.py --config config/omniscenes_cpo.ini --log log/cpo_test --method cpo
We also provide scripts for directly testing on your own data. First, prepare a query panorama image and 3D colored point cloud. Then, run the following command, and the renderings at the localized pose will be saved in the log directory.
python main.py --config config/omniscenes_cpo.ini --log LOG_DIRECTORY --method cpo --single --query_img PATH_TO_QUERY_IMG --color_pcd PATH_TO_COLORED_POINT_CLOUD
Run the following command for Stanford 2D-3D-S.
python main.py --config config/stanford_ldl.ini --log log/ldl_test --method ldl
Similarly, run the following command for OmniScenes
python main.py --config config/omniscenes_ldl.ini --log log/ldl_test --method ldl
We also provide scripts for directly testing on your own data. First, prepare a query panorama image and 3D colored point cloud. Then, extract 3D line segments using the following repository. Finally, run the following command, and the renderings at the localized pose will be saved in the log directory.
python main.py --config config/omniscenes_ldl.ini --log LOG_DIRECTORY --method ldl --single --query_img PATH_TO_QUERY_IMG --color_pcd PATH_TO_COLORED_POINT_CLOUD --line_pcd PATH_TO_LINE_CLOUD
Run the following command for Stanford 2D-3D-S.
python main.py --config config/stanford_fgpl.ini --log log/fgpl_test --method fgpl
Similarly, run the following command for OmniScenes
python main.py --config config/omniscenes_fgpl.ini --log log/fgpl_test --method fgpl
While the default setup is large-scale localization with multiple line maps, one can also test sinlge room localization with the follwoing command:
python main.py --config config/omniscenes_fgpl.ini --log log/fgpl_test --method fgpl --override "loc_exp_mode=single_room"
We also provide scripts for directly testing on your own data. First, prepare a query panorama image and 3D colored point cloud. Then, extract 3D line segments using the following repository. Finally, run the following command, and the renderings at the localized pose will be saved in the log directory.
python main.py --config config/omniscenes_fgpl.ini --log LOG_DIRECTORY --method fgpl --single --query_img PATH_TO_QUERY_IMG --line_pcd PATH_TO_LINE_CLOUD
Results from localization will be saved in .pkl files. To open a single .pkl file and print the recall at 0.1m, 5deg, run the following command.
python process_logger.py log/result.pkl 0.1 5
We also provide scripts for viewing aggregated results from multiple log files within a directory.
Suppose one has multiple log files whose paths start with the prefix ./log/ldl_test/scene_.
One can run the following command to view the aggregated recall at 0.1m, 5deg.
python aggregate_logger.py ./log/ldl_test/scene_ 0.1 5
If you find the repository useful, please cite the following papers:
@InProceedings{Kim_2021_ICCV,
author = {Kim, Junho and Choi, Changwoon and Jang, Hojun and Kim, Young Min},
title = {PICCOLO: Point Cloud-Centric Omnidirectional Localization},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2021},
pages = {3313-3323}
}@InProceedings{Kim_2022_ECCV,
author = {Kim, Junho and Jang, Hojun and Choi, Changwoon and Kim, Young Min},
title = {CPO: Change Robust Panorama to Point Cloud Localization},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
month = {October},
year = {2022},
pages = {176-192},
}@InProceedings{Kim_2023_ICCV,
author = {Kim, Junho and Choi, Changwoon and Jang, Hojun and Kim, Young Min},
title = {LDL: Line Distance Functions for Panoramic Localization},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2023},
pages = {17882-17892}
}@InProceedings{Kim_2024_CVPR,
author = {Kim, Junho and Jeong, Jiwon and Kim, Young Min},
title = {Fully Geometric Panoramic Localization},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2024},
}