Skip to content

Commit

Permalink
Update readme
Browse files Browse the repository at this point in the history
  • Loading branch information
aras-p committed Sep 14, 2023
1 parent 566bc31 commit bb6643a
Show file tree
Hide file tree
Showing 5 changed files with 21 additions and 15 deletions.
Binary file added Doc/shotAsset.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Doc/shotAssetCreator.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Doc/shotOverview.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed Screenshot01.jpg
Binary file not shown.
36 changes: 21 additions & 15 deletions readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Check out their website, source code repository, data sets and so on.

I've decided to try to implement the realtime visualization part (i.e. the one that takes already-produced gaussian splat "model" file) in Unity.

![Screenshot](/Screenshot01.jpg?raw=true "Screenshot")
![Screenshot](/Doc/shotOverview.jpg?raw=true "Screenshot")

The original paper code has a purely CUDA-based realtime renderer; other
people have done their own implementations (e.g. WebGPU at [cvlab-epfl](https://github.com/cvlab-epfl/gaussian-splatting-web), Taichi at [wanmeihuali](https://github.com/wanmeihuali/taichi_3d_gaussian_splatting), etc.).
Expand All @@ -15,25 +15,30 @@ Code in here so far is randomly cribbled together from reading the paper (as wel
- Splat color accumulation is done by rendering front-to-back, with a blending mode that results in the same accumulated color as their tile-based renderer.
- Splat sorting is done with a AMD FidelityFX derived radix sort, or (on DX11) with a GPU bitonic sort that is lifted from Unity HDRP codebase.

This is not a fast implementation yet!

## Usage

- Within Unity (2022.3), there's a `Scene.unity` that has a `GaussianSplatRenderer` script attached to it.
- The project defaults to DX12 on Windows, since then it can use a faster GPU sorting routine. DX11 should also work, at expense of performance.
- Metal and Vulkan also use the faster sorting approach.
- You need to point it to a "model" directory. The model directory is expected to contain `cameras.json` and
`point_cloud/iteration_7000/point_cloud.ply` inside of it.
- Since the models are quite large, I have not included any in this Github repo. The original [paper github page](https://github.com/graphdeco-inria/gaussian-splatting) has a a link to
[14GB zip](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/pretrained/models.zip) of their models.
- Press play.
- The gaussian splat renderer component inspector will have a slider to move the game view camera into one of the cameras from the model directory.
Or you can just move the game/scene view camera however you please.
- There are also various controls in the script to debug or visualize the data.
:warning: Note: this is all _**a toy**_, it is not robust, it does not handle errors gracefully, it does not interact or composite well with the "rest of rendering", it is not fast, etc. etc. Also, do not file bugs or issues just yet; I will most likely just ignore them and do whatever I please. I told you so! :warning:

First download or clone this repository and open as a Unity (2022.3) project. Note that the project defaults to DX12 on Windows,
since then it can use a faster sorting routine (DX11 should also work, but sorting will be slower).

Next up, **create some GaussianSplat assets**: open `Tools -> Create Gaussian Splat Asset` menu within Unity. In the dialog point it to
your gaussian splat "model" directory (it is expected to contain `cameras.json` and `point_cloud/iteration_7000/point_cloud.ply` inside of it,
and be an actual Gaussian Splat model, not something else). Pick desired compression options and output folder, and press "Create Asset" button.
If everything was fine, there should be a GaussianSplat asset that has a bunch of data images next to it.

Since the gaussian splat models are quite large, I have not included any in this Github repo. The original
[paper github page](https://github.com/graphdeco-inria/gaussian-splatting) has a a link to
[14GB zip](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/pretrained/models.zip) of their models.

In the game object that has a `GaussianSplatRenderer` script, **point the Asset field to** one of your created assets.
There are various controls on the script to debug/visualize the data, as well as a slider to move game camera into one of asset's camera
locations.

:warning: Note: this is all _**a toy**_, it is not robust, it does not handle errors gracefully, it does not interact or composite well with the "rest of rendering", it is not fast, etc. etc. I told you so!
_That's it!_

Wishlist that I may or might not do at some point:
- [ ] Make it respect the game object transform
- [ ] Make rendering faster (actual tiled compute shader rasterizer)
- [ ] Look at ways to make the data sets smaller (both on-disk and in-memory)
- [x] Integrate better with "the rest" of rendering that might be in the scene (BiRP)
Expand All @@ -44,6 +49,7 @@ Wishlist that I may or might not do at some point:

My own blog posts about all this _(so far... not that many!)_:
* [Gaussian Splatting is pretty cool!](https://aras-p.info/blog/2023/09/05/Gaussian-Splatting-is-pretty-cool/) (2023 Sep)
* [Making Gaussian Splats smaller](https://aras-p.info/blog/2023/09/13/Making-Gaussian-Splats-smaller/) (2023 Sep)

## Performance numbers:

Expand Down

0 comments on commit bb6643a

Please sign in to comment.