Use AbsGrad to get better results with less gaussians #3113

jb-ye · 2024-04-27T02:24:52Z

I am following the parameter setting as recommended in https://arxiv.org/pdf/2404.10484

In order to experiment this PR, one need to install the customized gsplat library as in this PR nerfstudio-project/gsplat#166

# uninstall previous gsplat and nerfstudio

pip install git+https://github.com/nerfstudio-project/gsplat.git@absgrad
pip install git+https://github.com/jb-ye/nerfstudio

I will follow up my experiments here. I recommend to experiment with one of the two settings (A, B) and use input image resolution higher than 1600 (e.g. 2k or 4k)

O splatfacto-big in latest main branch (--rasterize_mode antialiased)
A splatfacto-big of this PR (--rasterize_mode antialiased --densify_grad_thresh=0.0008)
B splatfacto-big of this PR (--rasterize_mode antialiased --densify_grad_thresh=0.0004)

O. baseline.
A. faster than B, high quality (expected to be better than O), smaller PLY asset.
B. slowest training, highest quality, largest PLY asset.

maturk · 2024-04-27T11:10:11Z

nerfstudio/models/splatfacto.py

@@ -80,6 +80,21 @@ def SH2RGB(sh):
    return sh * C0 + 0.5


+def resize_image(image: torch.Tensor, d: int):
+    """
+    Downscale images using the same 'area' method in opencv


Hi @jb-ye, just wondering do you have any reference for this idea? Would like to learn more, thanks.

I found that the absgrad is rather sensitive to how we downscale images during coarse-to-fine training. The previous implementation is not antialiased, and would introduce more noises in computing absgrad. The new downscale method in this PR is basically the same INTER_AREA method in opencv. This approach is actually recommended for most NeRF based approach for downscaling images.

actually OpenCV's INTER_AREA is a bit different because it also works with non-integer scales.

I would say this change is closer to skimage.transform.downscale_local_mean

ichsan2895 · 2024-04-28T13:10:26Z

Does this implementation support classic rasterizer? @jb-ye

ichsan2895 · 2024-04-28T15:03:00Z

I got an error when opening viewer when it trains

ns-train splatfacto-big --logging.steps-per-log 200 --vis viewer+wandb --viewer.websocket-port 7007 \
    --pipeline.model.rasterize-mode antialiased --pipeline.model.densify_grad_thresh=0.0008 \
    nerfstudio-data \
    --data path/to/scene --downscale-factor 1

It runs smoothly and I keep it until about 6000 iters, then I open viewer localhost:7007. It is not suddenly fail. I rotate the gaussians, got pretty result, and boom.. The training is broken.

The error:

Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/workspace/NERFSTUDIO_ABSGS/nerfstudio/nerfstudio/viewer/render_state_machine.py", line 222, in run
    outputs = self._render_img(action.camera_state)
  File "/workspace/NERFSTUDIO_ABSGS/nerfstudio/nerfstudio/viewer/render_state_machine.py", line 182, in _render_img
    scale = min(desired_depth_pixels / current_depth_pixels, 1.0)
ZeroDivisionError: division by zero

The dataset : Waterfall_Kopu

jb-ye · 2024-04-28T15:34:27Z

@ichsan2895 The viewer crashes sometimes and kills the training, but this is a separate issue: #3064

jb-ye · 2024-04-28T15:35:37Z

Does this implementation support classic rasterizer? @jb-ye

Yes

jb-ye · 2024-04-29T01:39:15Z

Some experiments done on bicycle dataset (It looks like our splatfacto-big is the new state-of-the-art!)

bicycle (downscale 2x):
main branch splatfacto-big
PSNR=24.62 SSIM=0.7256 LPIPS=0.2256
this PR splatfacto-big ( --rasterize_mode antialiased --densify_grad_thresh=0.0006)
PSNR=24.68 SSIM=0.7394 LPIPS=0.1861

bicycle (downscale 4x):
Zip-Nerf (according to https://jkulhanek.com/nerfbaselines/)
PSNR=25.867 SSIM=0.775 LPIPS=0.160
AbsGS (according to https://arxiv.org/pdf/2404.10484)
PSNR=25.373 SSIM=0.782 LPIPS=0.186
this PR splatfacto-big ( --rasterize_mode antialiased --densify_grad_thresh=0.0006)
PSNR=25.43 SSIM=0.7867 LPIPS=0.1451

jb-ye · 2024-04-29T16:18:44Z

Another mipnerf360 dataset

garden (downscale 4x):
Zip-Nerf (according to https://jkulhanek.com/nerfbaselines/)
PSNR=28.185 SSIM=0.863 LPIPS=0.085
AbsGS (according to https://arxiv.org/pdf/2404.10484)
PSNR=27.579 SSIM=0.870 LPIPS=0.104
main branch splatfacto-big
PSNR=27.4 SSIM=0.8645 LPIPS=0.083
this PR splatfacto-big ( --rasterize_mode antialiased --densify_grad_thresh=0.0006)
PSNR=27.57 SSIM=0.8702 LPIPS=0.078

kerrj · 2024-04-29T17:05:03Z

nerfstudio/models/splatfacto.py

@@ -103,7 +118,7 @@ class SplatfactoModelConfig(ModelConfig):
    """If True, continue to cull gaussians post refinement"""
    reset_alpha_every: int = 30
    """Every this many refinement steps, reset the alpha"""
-    densify_grad_thresh: float = 0.0002
+    densify_grad_thresh: float = 0.0008


have we verified that this threshold results in roughly the same psnr as the standard grad threshold?

This is the recommended setting in the paper. Based on my limited experiments so far, it creates less gaussians (~30% less) in splatfacto but comes with slightly better quality. So the training is faster in general.

kerrj · 2024-04-29T18:20:54Z

Just need to bump the gsplat version to 0.1.11 then looks good!

kerrj

lgtm!

abrahamezzeddine · 2024-05-13T20:00:32Z

I see that absgrad function has now been pushed to latest release. Do we have to enable it or is it configured by default when running ns-train splatfacto-big ? @jb-ye

Thank you!

jb-ye · 2024-05-14T13:53:14Z

@abrahamezzeddine it is used by default

jb-ye force-pushed the main branch from ec15a75 to 22b3729 Compare April 27, 2024 02:27

jb-ye mentioned this pull request Apr 27, 2024

Floater Free Gaussian Splats #3104

Closed

jb-ye force-pushed the main branch 2 times, most recently from 1bbb1e8 to 48c4eb1 Compare April 27, 2024 04:45

jb-ye marked this pull request as draft April 27, 2024 04:54

jb-ye force-pushed the main branch 2 times, most recently from 48c4eb1 to 3dc2056 Compare April 27, 2024 05:05

maturk reviewed Apr 27, 2024

View reviewed changes

ichsan2895 mentioned this pull request Apr 28, 2024

How to make gaussian splats as high quality as possible? #3115

Open

ichsan2895 mentioned this pull request Apr 29, 2024

Export Splatfacto-optimized poses #3116

Merged

kerrj reviewed Apr 29, 2024

View reviewed changes

jb-ye force-pushed the main branch from 59a65c7 to 291108c Compare April 29, 2024 17:54

jb-ye and others added 3 commits April 29, 2024 18:22

Use AbsGrad to get better results with less gaussians

7b70758

some adjustment such that splatfacto-big don't go OOM

e17d172

use new gsplat version

ebba43e

jb-ye force-pushed the main branch from 291108c to ebba43e Compare April 29, 2024 18:22

jb-ye marked this pull request as ready for review April 29, 2024 18:23

Merge branch 'main' into main

8eb1d46

kerrj approved these changes Apr 29, 2024

View reviewed changes

jb-ye merged commit b190874 into nerfstudio-project:main Apr 29, 2024
2 checks passed

bernard0047 mentioned this pull request May 20, 2024

Splatfacto training crashes after 18k steps when training on VR-NeRF dataset #3152

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use AbsGrad to get better results with less gaussians #3113

Use AbsGrad to get better results with less gaussians #3113

jb-ye commented Apr 27, 2024 •

edited

maturk Apr 27, 2024

jb-ye Apr 28, 2024

f-dy May 6, 2024

ichsan2895 commented Apr 28, 2024 •

edited

ichsan2895 commented Apr 28, 2024

jb-ye commented Apr 28, 2024 •

edited

jb-ye commented Apr 28, 2024

jb-ye commented Apr 29, 2024

jb-ye commented Apr 29, 2024

kerrj Apr 29, 2024

jb-ye Apr 29, 2024 •

edited

kerrj commented Apr 29, 2024

kerrj left a comment

abrahamezzeddine commented May 13, 2024

jb-ye commented May 14, 2024

Use AbsGrad to get better results with less gaussians #3113

Use AbsGrad to get better results with less gaussians #3113

Conversation

jb-ye commented Apr 27, 2024 • edited

maturk Apr 27, 2024

Choose a reason for hiding this comment

jb-ye Apr 28, 2024

Choose a reason for hiding this comment

f-dy May 6, 2024

Choose a reason for hiding this comment

ichsan2895 commented Apr 28, 2024 • edited

ichsan2895 commented Apr 28, 2024

jb-ye commented Apr 28, 2024 • edited

jb-ye commented Apr 28, 2024

jb-ye commented Apr 29, 2024

jb-ye commented Apr 29, 2024

kerrj Apr 29, 2024

Choose a reason for hiding this comment

jb-ye Apr 29, 2024 • edited

Choose a reason for hiding this comment

kerrj commented Apr 29, 2024

kerrj left a comment

Choose a reason for hiding this comment

abrahamezzeddine commented May 13, 2024

jb-ye commented May 14, 2024

jb-ye commented Apr 27, 2024 •

edited

ichsan2895 commented Apr 28, 2024 •

edited

jb-ye commented Apr 28, 2024 •

edited

jb-ye Apr 29, 2024 •

edited