Neural Surfaces

0. Setup

We build on top of the Neural_Radiance_Fields codebase. Copy the following files into your Neural_Surfaces codebase from the assignment 3 codebase:

dataset.py
sampler.py
ray_utils.py

1. Sphere Tracing

In this part we implement sphere tracing for rendering an SDF, and use this implementation to render a simple torus. We will need to implement the sphere_tracing function in a4/renderer.py. This function should return two outpus: (points, mask), where the points Tensor indicates the intersection point for each ray with the surface, and masks is a boolean Tensor indicating which rays intersected the surface.

You can run the code for part 1 with:

# mkdir images (uncomment when running for the first time)
python -m a4.main --config-name=torus

This should save part_1.gif in the `images' folder.

2. Optimizing a Neural SDF

In this part, we implement an MLP architecture for a neural SDF, and train this neural SDF on point cloud data. You will do this by training the network to output a zero value at the observed points. To encourage the network to learn an SDF instead of an arbitrary function, we will use an 'eikonal' regularization which enforces the gradients of the predictions to behave in a certain way (look up on the internet for hints).

In this part you need to:

• Implement a MLP tp predict distance: We populate the NeuralSurface class in a4/implicit.py. For this part, we need to define a MLP that helps you predict a distance for any input point. More concretely, we would need to define some MLP(s) in __init__ function, and use these to implement the get_distance function for this class. Hint: we can use a similar MLP to what we used to predict density in Neural_Radiance_Fields, but remember that density and distance have different possible ranges!

• Implement Eikonal Constraint as a Loss: Define the eikonal_loss in a4/losses.py.

After this, we should be able to train a NeuralSurface representation by:

python -m a4.main --config-name=points

This should save save part_2_input.gif and part_2.gif in the images folder. The former visualizes the input point cloud used for training, and the latter shows your prediction. We might need to tune hyperparameters (e.g. number of layers, epochs, weight of regularization, etc.) for good results.

3. VolSDF

In this part, we implement a function converting SDF -> volume density and extend the NeuralSurface class to predict color.

• Color Prediction: Extend the the NeuralSurface class to predict per-point color. We may need to define a new MLP (a just a few new layers depending on how you implemented Q2). You should then implement the get_color and get_distance_color functions.

• SDF to Density: Read section 3.1 of the VolSDF Paper and implement their formula converting signed distance to density in the sdf_to_density function in a4/renderer.py. In your write-up, give an intuitive explanation of what the parameters alpha and beta are doing here. Also, answer the following questions:

1. How does high beta bias your learned SDF? What about low beta?
2. Would an SDF be easier to train with volume rendering and low beta or high beta? Why?
3. Would you be more likely to learn an accurate surface with high beta or low beta? Why?

After implementing these, train an SDF on the lego bulldozer model with

python -m a4.main --config-name=volsdf

This will save part_3_geometry.gif and part_3.gif.

4. Phong Relighting

In this part, we'll be implementing the Phong reflection model in order to render the SDF volume we trained under different lighting conditions. In principle, the Phong model can handle multiple different light sources coming from different directions, but for our implementation we assume we're working with a single directional light source that is coming in from light_dir and is of unit intensity. We will feed in a dictionary of Phong parameters containing ks, kd, ka, n, which refer to the specular, diffuse, ambient, and shininess constants respectively. The specular, diffuse, and ambient components describe the ratio of reflection to the specular, diffuse, or ambient components of light. The shininess constant describes how smooth the surface is, with higher values making it smoother and thus shinier.

• Surface Normal Recovery: To relight our model, we only need to evaluate the surface normal of our volume to plug into our reflection model. This can be done by dividing the gradient by its norm and can be implemented in the get_surface_normal function.

• Reflection Model: Using the surface normals, implement the Phong reflection model in lighting_functions.py. To get the light direction, calculate the value in render_images in main.py.

Now, render the volume under different lighting using:

python -m a4.main --config-name=phong

This will save part_4_geometry.gif and part_4.gif in the images folder, showing your model under rotating lights.

5. Neural Surface Extras (CHOOSE ONE! More than one is extra credit)

5.1. Render a Large Scene with Sphere Tracing

In Q1, we rendered a (lonely) Torus, but to the power of Sphere Tracing lies in the fact that it can render complex scenes efficiently. To observe this, we try defining a ‘scene’ with many (> 20) primitives (e.g. Sphere, Torus, or another SDF from this website at different locations). Look up for equations of what the ‘composed’ SDF of primitives is. We can then define a new class in implicit.py that instantiates a complex scene with many primitives, and modify the code for Q1 to render this scene instead of a simple torus.

5.2 Fewer Training Views

In Q3, we relied on 100 training views for a single scene. A benefit of using Surface representations, however, is that the geometry is better regularized and can in principle be inferred from fewer views. Experiment with using fewer training views (say 20) -- you can do this by changing train_idx in data laoder to use a smaller random subset of indices). You should also compare the VolSDF solution to a NeRF solution learned using similar views.

5.3 Alternate SDF to Density Conversions

In Q3, we used the equations from VolSDF Paper to convert SDF to density. You should try and compare alternate ways of doing this e.g. the ‘naive’ solution from the NeuS paper, or any other ways that you might want to propose!