Postshot has better quality splats - due to suboptimal colmap usage/algorithm/both?

[tangfucius]

TL;DR
I compared the splats of a room generated by Splatfacto and Postshot with the same inputs, and consistently found Postshot to be crisper.

Do we already know if some components (ns-process-data or Splatfacto algorithm) are suboptimal, particularly regarding how ns-process-data uses colmap outputs?

Set up
Inputs (for the same room):

• iPhone 14 video, no special setting
• Budget android phone: 56 photos, using 1/60s shutter speed + ultrawide. The pictures aren't blurry.

Both Nerfstudio and Postshot were run on Windows.

Nerfstudio
I used ns-process-data video/images to generate inputs for training.

1. iPhone video: 364 images were extracted for the video, while transforms.json contains 342 frames.
2. android: transforms.json only contains 25 frames (out of 56), even if I used --matching-method exhaustive.

Postshot
I tried these 2 settings:

1. Import the raw video/images directly, and let Postshot register the camera poses (Use Best Images for image selection)
   • For iPhone, it tracked 219 images from the video
   • For android, it tracked 52/56 images
2. Use the cameras.bin + images.bin + points3D.bin + images folder generated from ns-process-data above
   • For iPhone, it managed to track 362/364 images
   • For android, it managed to track 47/56 images

Default settings were used for training (Splat MCMC). Subjectively, I found the splat quality of these 2 settings to be pretty similar.

Results
Postshot tend to have crisper results for all cases. All renders are 1920 x 1080.

iPhone
Nerfstudio

Postshot (direct import)

Postshot (colmap)

android
Nerfstudio

Postshot (direct import)

Postshot (colmap)

Questions
As a supporter of open source, I'd love to see Nerfstudio match the quality of Postshot. Asking these questions to better understand where I can possibly contribute:

1. Postshot matches more images using the same colmap outputs (the *.bin files) as Nerfstudio (transforms.json). Why is that the case?
2. Would hcol perform better than colmap for ns-process-data? Tried using it but haven't managed to get the pycolmap dependency to install on Windows so far.
3. Is the Splat MCMC algorithm used in Postshot an improved version of the original GS, or are they comparable? Would like to understand whether the quality difference mainly comes from SfM or the splatting algorithm - seems to be the former based on what I have seen so far.

[jb-ye]

Your iphone results doesn't look right to me. If you can share cameras.bin + images.bin + points3D.bin + images folder , I might be able to debug a bit.

[tangfucius]

Thanks for your response! I uploaded them here.

Is there anything else you'd suggest me to look at on my side? Bottom line, given the same inputs, do we expect the quality of Splatfacto to be similar to other implementations?

[ichsan2895]

Hello @tangfucius , it is not fair comparison since splatfacto uses some images for eval dataset and not all images trained.

You can maximize the power of Splatfacto by using all images for training.

Try to add --eval-mode fraction --train-split-fraction 1 flags.

For example:

ns-train splatfacto-big nerfstudio-data --data path/to/dataset --eval-mode fraction --train-split-fraction 1

[canxerian]

Did you discover the cause of this @tangfucius ?