Eval images metrics does not show in WanDB

[ichsan2895]

Eval images metrics does not show in WanDB. It just shows GPU memory, Train loss, Train iter, and ETA

Installation:

git clone --branch 8e0c68754b2c440e2d83864fac586cddcac52dc4 https://github.com/nerfstudio-project/nerfstudio
cd nerfstudio
pip install -e .

cd ../

git clone https://github.com/LingzheZhao/gsplat
cd gsplat
git submodule update --init --recursive
pip install -e .

cd ../

git clone https://github.com/WU-CVGL/BAD-Gaussians
cd BAD-Gaussians
pip install -e .

Run command:

ns-process-data images --data IMAGES --output-dir path/to/scene

ns-train bad-gaussians --logging.steps-per-log 200 --vis viewer+wandb --viewer.websocket-port 7007 \
    --data path/to/scene \
    --pipeline.model.camera-optimizer.mode "cubic" \
    --vis viewer+wandb \
    image-restore-data --downscale_factor 1

System information:

Ubuntu 22.04 LTS
Python 3.10
Pytorch 2.0.1+cu118

[LingzheZhao]

Sorry for the rambling. TL;DR: In the context of pose-optimizing NeRF/3D-GS, the novel-view evaluation requires an extra step after training, and I will release that script in a week (maybe). However, there are also some workarounds if you just want to see some deblurred images on the wandb.

---

Hi, it seems that you are running BAD-Gaussians on your custom dataset with the ImageRestorationDataparser.

FYI, we focus on two types of missions here, one is deblurring and the other is novel-view synthesis. For deblurring, ground-truth sharp images are required in the images_test folder for evaluation (written here), and that is only available with the synthetic datasets. While for the real-word datasets, we can only evaluate the novel-view synthesis.

The ImageRestorationDataparser works for the deblurring by default, so if images_test is not provided, there won't be any eval images, and the trainer will disable evaluation (written here). In this case, if you want to split the images into train & eval (i.e. novel-view synthesis), you can use the NerfstudioDataparser alternatively and pass these arguments after it, for example:

ns-train bad-gaussians --logging.steps-per-log 200 --vis viewer+wandb --viewer.websocket-port 7007 \
    --data path/to/scene \
    --pipeline.model.camera-optimizer.mode "cubic" \
    --vis viewer+wandb \
    nerfstudio-data --downscale_factor 1 \
    --eval-mode='interval' --eval-interval=8 \

Now it will use every $n^\textrm{th}$ frame for novel-view synthesis eval.

(Or you can copy the entire images folder and name the copy as images_test, and the ImageRestorationDataparser will treat them as ground-truth sharp images. Although the metrics are somehow meaningless, you can at least see the rendered sharp images pop up in the wandb.)

However, the evaluation metrics here should only be used as a reference and not as a final result - you will see the PSNR go up and then down. This is because the camera poses of the evaluation images comes from COLMAP, which are not accurate. While the camera poses of the training images are being optimized, the camera poses of the evaluation images are not.

(Also note that in our experiments, the novel-view images are sharp, unlike the blurry training images. If blurry images are configured to be eval images, it will not match our output in the evaluation (which will be deblurred sharp images, written here) and the metrics will be even less informative)

IMO, in a scene reconstruction pipeline, NeRF or 3D-GS, when you are optimizing the camera poses, which means the poses are not accurate, you can't use the poses from COLMAP to render the novel-view images - Instead, you should freeze the scene parameters and find the accurate pose for the novel-view eval image by optimizing it. At least, that's what we do in BAD-NeRF and BAD-Gaussians. Perhaps this is why many guys find that pose optimization decreases PSNR (nerfstudio-project/nerfstudio#2863) and (nerfstudio-project/nerfstudio#2891).

Unfortunately, I have little time to refactor the novel-view evaluation script at the moment as I'm working on some other projects. Right now it still looks like crap, maybe I'll refactor and release the script in a week.

P.S. I wrote the ImageRestorationDataparser at the beginning of the project, which doesn't seem to be useful now. Maybe I'll remove it later.

[ichsan2895]

Thank you for your detailed information. I will try this one later:

ns-train bad-gaussians --logging.steps-per-log 200 --vis viewer+wandb --viewer.websocket-port 7007 \
    --data path/to/scene \
    --pipeline.model.camera-optimizer.mode "cubic" \
    --vis viewer+wandb \
    nerfstudio-data --downscale_factor 1 \
    --eval-mode='interval' --eval-interval=8 \

[ZcsrenlongZ]

Thank you for your detailed information. I will try this one later:

ns-train bad-gaussians --logging.steps-per-log 200 --vis viewer+wandb --viewer.websocket-port 7007 \
    --data path/to/scene \
    --pipeline.model.camera-optimizer.mode "cubic" \
    --vis viewer+wandb \
    nerfstudio-data --downscale_factor 1 \
    --eval-mode='interval' --eval-interval=8 \

Have you generate a sharp 3DGS? I run the code on the synthetic dataset, but generate a blury 3DGS?

renders.mp4

[LingzheZhao]

@ichsan2895 @ZcsrenlongZ Apologies for I didn't test the functionality of this code in a fresh-new environment before. I switch to a different PC and installed nerfstudio, etc., from scratch, and reproduced the problems above:

• The camera poses are not being optimized, because camera_opt was not added to the param_groups;
• The metrics of the camera optimizer are missing, because they were not added to the model's training metrics.

My local nerfstudio was diverged with the upstream so I missed these lines. I believe these problems should be fixed in this commit. Sorry again for the inconvenience!

[ZcsrenlongZ]

@ichsan2895 @ZcsrenlongZ Apologies for I didn't test the functionality of this code in a fresh-new environment before. I switch to a different PC and installed nerfstudio, etc., from scratch, and reproduced the problems above:

* The camera poses are not being optimized, because `camera_opt` was not added to the `param_groups`;

* The metrics of the camera optimizer are missing, because they were not added to the model's training metrics.

My local nerfstudio was diverged with the upstream so I missed these lines. I believe these problems should be fixed in this commit. Sorry again for the inconvenience!

Yes! It works! Thanks for your help!

renders.mp4

[ichsan2895]

Why it does not work with me...

# clean up previous version
pip uninstall gsplat bad-gaussians

git clone --branch 8e0c68754b2c440e2d83864fac586cddcac52dc4 https://github.com/nerfstudio-project/nerfstudio
cd nerfstudio
pip install -e .

pip install git+https://github.com/LingzheZhao/gsplat

cd ../

git clone https://github.com/WU-CVGL/BAD-Gaussians
cd BAD-Gaussians
pip install -e .

This is the latest bad-gaussians commit = 25155be

The code for running Dblur-nerf dataset:

>> ns-train bad-gaussians \
    --data /workspace/BAD-GAUSSIAN/data/real_camera_motion_blur/blurparterre \
    --pipeline.model.camera-optimizer.mode "cubic" \
    --vis viewer+wandb \
    deblur-nerf-data \
    --downscale_factor 4

Traceback (most recent call last):
  File "/usr/local/bin/ns-train", line 5, in <module>
    from nerfstudio.scripts.train import entrypoint
  File "/workspace/NERFSTUDIO_v102/nerfstudio/nerfstudio/scripts/train.py", line 62, in <module>
    from nerfstudio.configs.method_configs import AnnotatedBaseConfigUnion
  File "/workspace/NERFSTUDIO_v102/nerfstudio/nerfstudio/configs/method_configs.py", line 724, in <module>
    all_methods, all_descriptions = merge_methods(all_methods, all_descriptions, *discover_methods())
  File "/workspace/NERFSTUDIO_v102/nerfstudio/nerfstudio/plugins/registry.py", line 43, in discover_methods
    spec = discovered_entry_points[name].load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/workspace/BAD-GAUSSIAN/BAD-Gaussians/bad_gaussians/bad_config_method.py", line 12, in <module>
    from bad_gaussians.image_restoration_dataparser import ImageRestorationDataParserConfig
ModuleNotFoundError: No module named 'bad_gaussians.image_restoration_dataparser'

I also get similar error with ns-train bad-gaussians -h. Already report it in this thread = ns-train bad-gaussians -h gives error

[LingzheZhao]

Hi, I also left a comment there. Sry this is caused by recent remove of image_restoration_dataparser, as it may have caused some confusion in #3. You can first uninstall the bad_gaussians:

pip uninstall bad_gaussians

Then reinstall it, the problem should be gone.
Maybe bad_gaussians was typoed into bad-gaussians and thus it was not uninstalled? If this still doesnot work, maybe also try reinstalling nerfstudio with it, this should clear all the entry points...

(This only happens if you have installed it before. entry_point of image_restoration_dataparser was registered and needs to be removed by this uninstallation)

[ichsan2895]

Thanks @LingzheZhao

I had success with this fix:

pip uninstall gsplat nerfstudio bad-gaussians

pip install git+https://github.com/LingzheZhao/gsplat.git@f62f4132351a65ee4552dd0a42212b86dd90036f

# If it works, dont touch it. I am afraid the newest commit of nerfstudio caused new problem
pip install git+https://github.com/nerfstudio-project/nerfstudio.git@ae6c46cfeaebbe28f9cd48fa986755e27b5f0ae2

# latest commit as 28th March 2024
pip install git+https://github.com/WU-CVGL/BAD-Gaussians.git@5f664ef26e527a9e6bca47ef14bd0ee461bbde7c