Inquiry on Applying IDA to Generated Rays in NeRF-based Models

[SPA-junghokim]

I hope you're doing well. I'm writing to express my appreciation for your research on NeRF and its rendering techniques, particularly the application to OCC models. Your work has inspired my project, which explores integrating camera characteristics more effectively into model structures.

I have a specific question regarding the ray generation process when applying IDA. Should post-rotation and post-translation adjustments be applied to the generated rays after projection onto images? Understanding the implications of IDA on ray adjustments would greatly benefit my research.

I look forward to your insights and thank you in advance for your guidance.

Best
Jungho Kim

[pmj110119]

Hi Kim, I believe it's unnecessary.

Rays are in the world coordinate system, not the camera coordinate system, so the IDA transformation applied at the image level doesn't affect rays. IDA does impact the process of converting 2D image features to 3D volume features, requiring careful handling, but it doesn't affect rays.
For a ray generated from the original image, you simply need to directly retrieve color, class, depth, and other pixel label values from the corresponding pixel. This process doesn't involve coordinate transformations.

[SPA-junghokim]

I really appreciate your reply. I understood your word.

Regards,
Jungho Kim

[SPA-junghokim]

I have another question: In the nerf_head, using the rendering loss versus not using it results in about a 7-fold difference in training time. Is it normal for it to take significantly longer?

[pmj110119]

7x increase in time seems somewhat abnormal. Could you provide the time cost of one iteration, and is it similar to that observed in the example log?

[SPA-junghokim]

It has been confirmed to be a hardware issue with our own server. Thank you.

I have one more question:
Do we have to use separate heads for density and semantic, or can we treat the output from a single head, like in BEVDET-Occ, as density volume for areas not predicted as empty-class and render them?

[pmj110119]

You can of course use output predicted by single head and then separate channels to get density and semantic logits.

However, in our early experiments, the performance of using two MLPs to predict geometry (density) and semantics respectively was slightly better, but this has not been rigorously verified.