Equirectangular (360) workflows and the Rig Bundle Adjuster

[felixgeen]

I have been trying to workout how to get data from 360 cameras (Theta/Insta360) into Instant NGP. Since FISHEYE support was added I tried that first but with very poor results. it seems that loading the unstitched 180 degree perspectives from the camera straight into Instant NGP does not work and creates very poor results.

The second technique I've tried, is stitching the 2 fisheye results into Equirectangular images and then splitting those into 5-7 different perspective views.

The issue I'm having is during alignment in COLMAP it is very hard to keep the images bundled together in their correct rig. I would love to be able to use the Rig Bundle Adjuster feature which is a part of COLMAP but I cant workout how to initialise it within colmap2nerf.py.

In ideal circumstances COLMAP manages to align most of the images correctly with only a slight amount of error, often missing roughly 30% of the dataset. In this image you can see it has missed the down facing images on most of the rig bundles.

I have been using Agisoft for alignment but it produces a different set of issues unrelated to rig alignment.

Any help would be appreciated in getting COLMAP to work with camera rigs like this. I'm happy to upload my dataset for people to experiment with also.

Best,
Felix

[Tom94]

Hi, I added support for directly training from equirectangular images earlier today -- it might be worth a shot if have the orientation of your equirectangular image. Simply add the key "equirectangular": true to transforms.json to enable it. (Similarly, latlong images are supported via "latlong": true.)

[felixgeen]

Hi @Tom94 ,

I decided to try and align the equirectangular images by first converting the dataset from Equirectangular into single Pinhole camera projections like this:

Next I aligned all of those images in metashape:

Then I ran the metashape > nerf script which gives me all those images correctly aligned and orientated in a tranforms.json. Then i just swapped the filenames so that the JSON points to the original equirectangular images (eg R0012035-1_yaw0.jpg becomes R0012035-1.jpg) the JSON file looks like this:

{
    "camera_angle_x": 2.0984635591937626,
    "camera_angle_y": 2.0984635591937626,
    "fl_x": 1103.3109939571805,
    "fl_y": 1103.3109939571805,
    "k1": 0.004089683855301384,
    "k2": -0.005282626103035978,
    "p1": -6.774483181337844e-05,
    "p2": 0.0006778244853053426,
    "cx": 1919.3324283469801,
    "cy": 1910.9120952764267,
    "w": 3840.0,
    "h": 3840.0,
    "aabb_scale": 16,
    "frames": [
        {
            "file_path": "./images/R0012035-1.jpg",
            "sharpness": 18.870548027039874,
            "transform_matrix": [
                [
                    0.003860656726075219,
                    0.580960546214069,
                    0.8139225633144066,
                    0.7760623968474493
                ],
                [
                    0.9699329120690865,
                    -0.20023983458720032,
                    0.1383262618943797,
                    -0.003241985981361606
                ],
                [
                    0.24334182011077599,
                    0.7889162518209026,
                    -0.5642658116508863,
                    -0.5756413392497797
                ],
                [
                    0.0,
                    0.0,
                    0.0,
                    1.0
                ]
            ]
        },

etc etc etc for all 32 images in the dataset

What do I need to change here in the JSON file to make these equirectuanglar images work properly instead of the pinhole projections? You mentioned

Simply add the key "equirectangular": true to transforms.json to enable it.

But I think I would need to modify more than that with this hack-around workflow.

Ive uploaded this dataset and the tranforms.json file in case anyone would like to play with it: https://drive.google.com/file/d/1fz2cPxs4s42M52fIXwSN99wJ_6AOPfgR/view?usp=share_link
(this dataset was made by my friend Josha Eiffel)

[yusuf-wadi]

If I run your dataset on instant-ngp with equirectangular set to true, I do achieve a result, but not exactly the highest quality result. As in, I am able to traverse the scene, albeit, a low quality rendition of the scene. I have scene evidence online of people achieving high quality results with 360 video as input, but I am still unsure how they did it exactly.

example:
https://www.youtube.com/watch?v=IG-Bo-2CEGI&t=13s

[adriano303]

Hello @Tom94, I have the same problem, I managed to get the camera positions and orientation but I didn't understand how to add the key "equirectangular": true to the .json file, btw do I need to change all the parameters below? most of them loose meaning for an equirectangular footage.
"camera_angle_x":
"camera_angle_y":
"fl_x":
"fl_y":
"k1":
"k2":
"p1":
"p2":
"cx":
"cy":

[felixgeen]

I’m pretty sure all the folks who are uploading these “360 camera nerf” videos are using the trick I mentioned above where you split the equirectangular projection into a bunch of regular pinhole/flat projection fov style images and then align all those. It’s super inefficient and clunky. It would be awesome if Tom94 could practically explain how to use the equirectangular projection option he added. Perhaps adding an additional equirectangular example dataset to the GitHub? :)

[KotaYonezawa]

Hi All,
I tried instant NeRF using equirectangular images by RICOH Theta camera, converted using Theta App.
In my trial, following transforms.json description worked.
(Camera alignment was done by Metashape.)

{
"latlong": "true",
"camera_angle_x": 6.283185307,
"aabb_scale": 16,
"scale": 0.033,
"offset": [
0.5,
0.5,
0.5
],
"frames": [
...

Not "equirectangular" but "latlong" seems correct for common panorama equrectangular images.
"camera_angle_x" parameter is dummy.
Without "camera_angle_x" parameter, instant NGP returns error. so I added it.
It seems that the value of "camera_angle_x" has no effect now.
aabb_scale/scale/offset parameters should be changed depend on the scene.

But in my trial, the quality of NeRF rendering result of "latlong" setting is worse than using regular flat projection images converted from same equirectangular image. I'm not sure why...
If someone has additional information, please let me know...

Rendering result using "latlong" setting (original equirectanglar image is used for instant NeRF input)

Rendering result using plane images. (13 plane images (FoV = 90 degree) are generated from one equirectanglar image, and used for instant NeRF input)

[khalidmaquilang]

Hello I am very intrigue on how you use nerf in Equirectangular images. May i know script should I use? I have already images.. what will be my next step..

[yusuf-wadi]

when you complete COLMAP processing, go into the transforms json file and and you can add the line "equirectangular": true