A perhaps great, perhaps stupid idea: more than one ideas to extract mesh from GS.

[yuedajiong]

1. add regulirization items, force: a) algin points to surface, b)flatten every point。 (refer: SuGaR https://arxiv.org/abs/2311.12775)

2. learn a mapping fucction, mesh = f_gs_to_mesh(gs_points_convergent, [cameras_in_trainset, target_images_in_trainset])
   on this direction, I suppose we can learn the regular pattern(intuitively, there should be), that is better than a hard-code algorithm(maybe difficult) which do the mapping.i

great-masters, any comment?

[mochou-wujiu]

if you want to train the fuc, where does the trainning data come from? perhaps train 3d gaussian on multiple 3d datasets? there's another issue about the gaussian itself, there're too many gaussians in a well-trained scene, and the nums vary across scenes.

[yuedajiong]

@mochou-wujiu
yes, this fucntion is very difficult to get; but once we got, it is very useful.
and, let me discuss HOW:

1. trian official GS on different dataset(objects, scenes), even different hyper-parameters, and we get: GS_official
   A.1 we can train SuGaR similar with step#1, then we get: "GS_suguar"
   A.2 the first way, we can train the function: GS_aligned_likes_sugar = (GS_official, ...any others useful infos: [cameras_in_trainset, target_images_in_trainset]...) . the GS_aligned_likes_sugar is approximately equal to surface/mesh.
   B.1 train on a huge dataset with mesh.
   C.1 we can still train by unsupervised-style. input-images -> (official, can be pre-trained)guass-points --(here:important)-> mesh --> nvdiffrast --> prime-images.

anyway, we need that mesh = f(gauss-points, ...)
we "assume" that there is an implicit mapping function from GS that is not strongly related to the surface to the object real surface, which can be learned from large amounts of data.

btw: if we use some surface-constraint losses(include but not limit: sugar) and other trickes(eg. mask), maybe it should be easier.