Improving Montmajour meshes - H100

[rebecca-lay3rs]

Improving Montmajour meshes - H100

Problem 1: Triangle vertices

• Change interpolation method in blender: https://docs.blender.org/manual/en/latest/render/shader_nodes/textures/image.html from linear to closest pixel (Question about how to remove the verticies lines. Anttwo/SuGaR#119)

Problem 2: Filling up empty spaces in mesh

• mainly behind the object

n_points_in_range = 21

# We compute ray samples
points_range = torch.linspace(-range_size, range_size, n_points_in_range).to(self.device).view(1, -1, 1)  # (1, n_points_in_range, 1)
points_range = points_range * points_stds[..., None, None].expand(-1, n_points_in_range, 1)  # (n_points, n_points_in_range, 1)
samples = (all_world_points[:, None, :] + points_range * camera_to_samples[:, None, :]).view(-1, 3)  # (n_points * n_points_in_range, 3)
samples_closest_gaussians_idx = closest_gaussians_idx[:, None, :].expand(-1, n_points_in_range, -1).reshape(-1, self.knn_to_track)

Problem 3: Background:

• Idea 1) removing the points in the point cloud that we dont want! -> too complicated to do in COLMAP, would have to rewrite multiple files so lets put this in pause

• Idea 2) removing the background of the objects in every photo of the dataset and inserting it in COLMAP to obtain the point cloud only around the object, without the points of the background:

folder: Cloitre_Gal1-2_A2_sheep_tests
command: python train.py -s data/Cloitre_Gal1-2_A2_sheep_tests/ -c outputs_GS/Cloitre_Gal1-2_A2_sheep_tests/ -r "sdf" --low_poly True --poisson_depth 8 --experiment_name Cloitre_Gal1-2_A2_sheep_tests

	with background	without background
images
point3D.ply
mesh.obj text
mesh.obj gray
side view

conclusion: Sugar struggles to treat objects without background BUT the side part is better defined without the background

• Idea 3) Input the points3D.ply without background on Sugar but input the images with background (sort of mixed dataset)

folder: Cloitre_Gal1-2_A2_sheep_tests_2
command: python train.py -s data/Cloitre_Gal1-2_A2_sheep_tests_2/ -c outputs_GS/Cloitre_Gal1-2_A2_sheep_tests/ -r "sdf" --low_poly True --poisson_depth 8 --experiment_name Cloitre_Gal1-2_A2_sheep_tests_2

the object is behind the background

conclusion: does not work, Sugar adds the background in front of the object

[rebecca-lay3rs]

• Idea 4) Use GroudingDino + SAM to segment better the input dataset and put it in COLMAP
  using GDINO+SAM
  https://github.com/IDEA-Research/GroundingDINO
  https://github.com/facebookresearch/segment-anything

installation using Docker here: https://github.com/Lay3rss/Grounded-Segment-Anything

how to use mask_path COLMAP
colmap/colmap#1016 -> how to use mask_path
colmap/colmap#522 -> how to use mask_path
https://colmap.github.io/faq.html#mask-image-regions
colmap/colmap#954
colmap/colmap#416 -> Filter outlier reconstructed points?
colmap/colmap#1251
colmap/colmap#1460
colmap/colmap#208

• Idea 5) Use GroudingDino + SAM to segment better the input dataset and put it in Gaussian Splatting directly

render without background
graphdeco-inria/gaussian-splatting#542
graphdeco-inria/gaussian-splatting#127
forked repo from GS with mask implementation: https://github.com/yzslab/gaussian-splatting

./output/coarse/Cloitre_Gal1-2_A2_sheep_masks/sugarcoarse_3Dgs7000_sdfestim05_sdfnorm02/15000.pt
Mesh output path: ./output/coarse_mesh/Cloitre_Gal1-2_A2_sheep_masks
Surface levels:
[0.3]
Decimation targets:
[200000]
Use custom bbox: False
Use eval split: True
GPU: 0
Use centers to extract mesh: False
Use marching cubes: False
Use vanilla 3DGS: False

Loading the initial 3DGS model from path outputs_GS/Cloitre_Gal1-2_A2_sheep_masks/...
Warning: image extension .jgp not supported.
Traceback (most recent call last):
File "/root/SuGaR/train.py", line 163, in
coarse_mesh_path = extract_mesh_from_coarse_sugar(coarse_mesh_args)[0]
File "/root/SuGaR/sugar_extractors/coarse_mesh.py", line 129, in extract_mesh_from_coarse_sugar
nerfmodel = GaussianSplattingWrapper(
File "/root/SuGaR/sugar_scene/gs_model.py", line 111, in init
cam_list = load_gs_cameras(
File "/root/SuGaR/sugar_scene/cameras.py", line 126, in load_gs_cameras
mask=resized_mask ,) ####################
UnboundLocalError: local variable 'resized_mask' referenced before assignment