Allow gaussian splatting to initialize start points from other data pipelines than COLMAP

[kerrj]

Currently, gaussian splatting intakes a pointcloud for initialization either through: 1) using a COLMAP dataset, or 2) adding a pointcloud to transforms.json and using the nerfstudio dataparser to read it. Currently, data intake pipelines like polycam, metashape, etc do not output a pointcloud and thus gaussian splatting will use a random initialization of points, which produces much worse results.

To fix this, we should edit the data intake scripts to add point clouds to the output transforms.json files and use the nerfstudio dataparser to read them. #2557 added the capability to load .ply files in the nerfstudio dataparser

• Add to the COLMAP data processing pipeline to save pointclouds of the SFM points as output, and add it to the transforms.json
• Switch to nerfstudio dataparser as the default for splatting
• implement pointcloud saving in other data processing pipelines

[jb-ye]

FYI, metashape has a script for exporting colmap dataset (undistorted images): https://github.com/agisoft-llc/metashape-scripts/blob/master/src/export_for_gaussian_splatting.py

[FreakTheMighty]

@kerrj is it your experience that the guassian-splat pipeline runs, just with reduced quality, when using other data sources? What I'm seeing ATM is that when using polycam as input an error is thrown.

    assert colmap_path.exists(), f"Colmap path {colmap_path} does not exist."
AssertionError: Colmap path Garage/colmap/sparse/0 does not exist.

[Tekn1ck]

I am also getting this when using RealityCapture. It would be super to get this up and running. If we could export the point cloud that Reality Capture generates would that help?

assert colmap_path.exists(), f"Colmap path {colmap_path} does not exist."

AssertionError: Colmap path C:\Users\1user\nerfstudio\data\nerfstudio\farmhouse\colmap\sparse\0 does not exist.

(nerfstudio) C:\Users\Tekuser\nerfstudio>ns-train nerfacto --data C:\Users\1user\nerfstudio\data\nerfstudio\farmhouse
[15:24:27] Using --data alias for --data.pipeline.datamanager.data

[brentyi]

I think these errors are because the default dataparser for gaussian-splatting is set to colmap, while all other models set it to nerfstudio-data. There are plans to transition splatting over to the latter.

Specifying the nerfstudio-data subcommand should help fix assert colmap_path.exists() errors: ns-train gaussian-splatting --data /path/to/a/processed/dataset nerfstudio-data

[Tekn1ck]

That worked and I was able to train a Gaussian splat! Next issue I have is outputting the splat to PLY I get this error below. Would anyone know how to solve this?

(nerfstudio) C:\Users\1user\nerfstudio>ns-export gaussian-splat --load-config C:\Users\1user\nerfstudio\outputs\farmhouse\gaussian-splatting\2024-01-19_003623\config.yml --output-dir C:\Users\1user\nerfstudio\outputs\farmhouse\gaussian-splatting\2024-01-19_003623\PLY
Traceback (most recent call last):
File "C:\Users\1user\miniconda3\envs\nerfstudio\lib\runpy.py", line 194, in run_module_as_main
return run_code(code, main_globals, None,
File "C:\Users\1user\miniconda3\envs\nerfstudio\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\1user\miniconda3\envs\nerfstudio\Scripts\ns-export.exe_main.py", line 4, in
File "C:\Users\1user\nerfstudio\nerfstudio\scripts\exporter.py", line 39, in
from nerfstudio.exporter import texture_utils, tsdf_utils
File "C:\Users\1user\nerfstudio\nerfstudio\exporter\texture_utils.py", line 26, in
import mediapy as media
File "C:\Users\1user\miniconda3\envs\nerfstudio\lib\site-packages\mediapy_init.py", line 130, in
import IPython.display
File "C:\Users\1user\miniconda3\envs\nerfstudio\lib\site-packages\IPython_init.py", line 53, in
from .terminal.embed import embed
File "C:\Users\1user\miniconda3\envs\nerfstudio\lib\site-packages\IPython\terminal\embed.py", line 15, in
from IPython.core.interactiveshell import DummyMod, InteractiveShell
File "C:\Users\1user\miniconda3\envs\nerfstudio\lib\site-packages\IPython\core\interactiveshell.py", line 73, in
from IPython.core.history import HistoryManager
File "C:\Users\1user\miniconda3\envs\nerfstudio\lib\site-packages\IPython\core\history.py", line 11, in
import sqlite3
File "C:\Users\1user\miniconda3\envs\nerfstudio\lib\sqlite3_init_.py", line 23, in
from sqlite3.dbapi2 import *
File "C:\Users\1user\miniconda3\envs\nerfstudio\lib\sqlite3\dbapi2.py", line 27, in
from _sqlite3 import *
ImportError: DLL load failed while importing _sqlite3: The specified module could not be found.

[kerrj]

Update on this.... items 1 and 2 are done thanks to @CardiacMangoes, it should be pretty straightforward (but a bit tedious) to update other pipelines to output a pointcloud .ply file and add it to the transforms.json

[kerrj]

#3122 adds metashape!

[jb-ye]

#3131 adds realitycapture and fixed existing bugs in calculating cx,cy!

[SchmidL]

Hi - short question:

So I use the https://github.com/agisoft-llc/metashape-scripts/blob/master/src/export_for_gaussian_splatting.py to export the metashape project and then wanted to preprocess it with #3122 (comment)
but i need additionally to export the ply point cloud, or? I probably will export the dense model as ply

[jb-ye]

@SchmidL using sparse point cloud should be okay.

[SchmidL]

Perfect! But sorry for asking - what is right now the suggested pipeline out of Metashape and/or Reality Capture?
After the finished processing with the linked script creates already .bin files, but I cannot relate that to any documented pre/processing.
Thank you very much for any up-to-date guidance or a documention.

[jb-ye]

@SchmidL Use ns-process-data to convert metashape / RC exports into nerfstudio format (i.e. #3122 (comment) and #3131 (comment)), and train with the nerfstudio formattedd data.

[kxwangzju]

@SchmidL Use ns-process-data to convert metashape / RC exports into nerfstudio format (i.e. #3122 (comment) and #3131 (comment)), and train with the nerfstudio formattedd data.

@jb-ye Hi, I am wondering how other open-sourced Gaussian Splatting methods can be used with nerfstudio formatted data. Quite a lot of them use input in COLMAP data format. Are there any methods that can convert nerfstudio format into COLMAP? I did do some searching but I cannot find one. Or just write a dataloader for those methods? Thank you for your time.