Missing 'render' in 'nr3d_lib'

[rikkazz]

At Lightning-NeRF/lightning_nerf/sampler.py
from nr3d_lib.render.pack_ops import merge_two_packs_sorted_aligned
from nr3d_lib.render.raysample import packed_sample_pdf

can not find 'render' in nr3d_lib v0.6.0

I see that nr3d_lib==0.3.1 is required, but I can not find the release version in https://github.com/PJLab-ADG/nr3d_lib/tree/main/nr3d_lib

Is this a special version？

[rikkazz]

never mind, replaced by
from nr3d_lib.graphics.pack_ops import merge_two_packs_sorted_aligned
from nr3d_lib.graphics.raysample import packed_sample_pdf

[XJay18]

Hi, sorry for the late reply. Since nr3d_lib has recently had a major code update, you may need to use a previous version of nr3d_lib. As mentioned in the README, we have tested nr3d_lib with version 0.3.1 (more specifically, commit id: e4eba51). Thus, if you do not want to modify the code, you may checkout this commit in nr3d_lib.

I am also grateful for your mentioning how to modify the import script to align with the updated nr3d_lib. Thank you for your time.

[szhang963]

Hi, why do we use nr3d_lib in this project? Can it be replaced by some modules in NeRFStudio?
Thanks in advance.

[XJay18]

@szhang963
Hi, thank you for your interest.

Briefly, our sampling process (to generate sample points along rays) consists of two steps:

1. Sampling guided by an occupancy grid.
2. (Optionally) Perform importance sampling based on previous sampled results, similar to the fine stage of the hierarchical sampling used in Vanilla NeRF.

The second round of importance sampling may assist in sampling near surface areas. If the second round of importance sampling is disabled, then nr3d_lib is not required.

• You may set --pipeline.model.pdf-num-samples-per-ray 0 in the training script to disable the importance sampling.
• Then, you may comment out the import of nr3d_lib used in lightning_nerf/sampler.py.

The reason is as follows.

This project uses an occupancy grid sampler to sample points along rays, resulting in different numbers of samples per ray. Therefore, we cannot use conventional representations like (num_rays_per_batch, num_samples_per_ray, ...) used in NeRFStudio to perform a second round of importance sampling. Instead, the output of occupancy grid sampler is represented as (num_samples_per_batch, ...) along with an index to indicate which ray the sample belongs to. nr3d_lib supports a range of operations for this representation. Thus, we made a dependency on nr3d_lib.

[szhang963]

@XJay18 Thank you for your detailed reply.
What do the render results change if the importance sampling is disabled?

[XJay18]

@szhang963
Basically, turning off the importance sampling would have a slight effect on interpolation metrics. However, the extrapolation results would be inferior since extrapolation requires a more accurate sampling around true surfaces.

Please see the following illustration (especially lane markings) of an extrapolation experiment for an intuitive understanding.

[szhang963]

@XJay18 Thanks for your interpretation in detail. I will try it.