Can nerfacc support mipnerf and mipnerf360?

[CSU-NXY]

Hi, thanks for your great framework. I'm trying to use nerfacc to accelerate mipnerf and mipnerf360, but this seems harder than I thought, since these two models sample cones instead of points, and need compute covariance after sampling.

I'll be very appreciate it if you can give me some hints!

[liruilong940607]

Hi we took into consider the mip-nerf formulation when designing nerfacc, so it should be fairly easy to work with mipnerf.

Specifically, for mipnerf, each ray would have origins, directions, and radii:

rays_o: Tensor = ...  # ray origins. (n_rays, 3)
rays_d: Tensor = ...  # ray normalized directions. (n_rays, 3)
rays_radii: Tensor = ... # ray radius on image plane. (n_rays,)

And you can compute the covariance within the sigma_fn you defined:

def sigma_fn(
   t_starts: Tensor, t_ends:Tensor, ray_indices: Tensor
) -> Tensor:
   """ Query density values from a user-defined radiance field.
   :params t_starts: Start of the sample interval along the ray. (n_samples, 1).
   :params t_ends: End of the sample interval along the ray. (n_samples, 1).
   :params ray_indices: Ray indices that each sample belongs to. (n_samples,).
   :returns The post-activation density values. (n_samples, 1).
   """
   t_origins = rays_o[ray_indices]  # (n_samples, 3)
   t_dirs = rays_d[ray_indices]  # (n_samples, 3)
   t_radii = rays_radii[ray_indices] # (n_samples,)
   mean, cov = cylinder_to_gaussian(t_starts, t_ends, t_origins, t_dirs, t_radii)
   sigmas = radiance_field.query_density(mean, cov)
   return sigmas  # (n_samples, 1)

Same for the sigma_rgb_fn.

[liruilong940607]

For theOccupancyGrid to work with mipnerf, the occ_eval_fn can be defined in such way (pesudo code):

def occ_eval_fn(x):
    # Randomly sample cameras for each point. we want the
    # occupancy grid to cache the maximum possible opacity at
    # location x. For mip-nerf, the density is "camera-dependent".
    camera_ids = torch.randint(
        0, len(train_dataset), (x.shape[0],), device=device
    )
    cameras = train_dataset.cameras[camera_ids]
    # compute some infos based on the camera.
    t_origins, t_dirs, t_radii = compute_sample_infos(cameras, x)
    # compute sample interval based on the camera.
    t_starts, t_ends = compute_sample_interval(cameras, x)
    # compute mean, cov of the samples.
    mean, cov = cylinder_to_gaussian(t_starts, t_ends, t_origins, t_dirs, t_radii)
    # compute occupancy
    density = radiance_field.query_density(mean, cov)
    return density * (t_ends - t_starts)

For your reference, there is a demo of compute camera-dependent step_size:

https://github.com/KAIR-BAIR/nerfacc/blob/107d7466ef3fcfadd64e8b57ad3c1aafd4874fe0/examples/train_ngp_nerf.py#L190-L213

For mip-nerf, it would be a bit more complicated but the idea is the same.

[CSU-NXY]

Thank you very much!

I have one more question for updating occupancy grid. In occ_eval_fn, instead of randomly sampling cameras to construct a conical frustum for each point, can we just compute the mean and cov of the voxel grid?

In my mind, mipnerf use Multivariate Gaussian distribution to simulate the coordinate distribution inside a conical frustum. Since we are focusing voxel grid (or cube) when updating OccupancyGrid, it might be reasonable to use the mean and cov of coordinates distribution inside this grid directly.

[liruilong940607]

Hi!

I think that is an interesting point. It might be ok to just use the mean and cov for the grid cell. But in that case you are essentially use the OccupancyGrid to express the average occupancy within each grid cell.

Though the way we use OccupancyGrid during marching is that we expect it to store the maximum possible occupancy at each cell, so we can skip the cell safely if we find its occupancy is below a certain threshold.

Anyway I would be very curious to see how things going with mipnerf if you plan to try it out!

[CSU-NXY]

Thank you!

I think you are right. The average occupancy might not be accurate enough to indicate which grid cell can be skipped. This is just an idea came to me when reading your code snippet and I'm going to try the method you provided first.