Linear or sRGB？

[xmk2222]

I tried training nerf with my own dataset and it worked well, but I'm quite confused about the color space. I found 3 color space settings(python api) in i-ngp which I can't fully understand:

• testbed.color_space: default=linear
• testbed.nerf.training.linear_colors: default=false
• testbed.render(..., linear, ...): default=true

The questions are:

1. The meaning of the settings above？
2. The color space of the target images and the corresponding settings?
3. The color space of the rendered images?
4. The difference between training in the sRGB or the linear color space?
5. If I would like to turn on the optimize_exposure, what the settings should be like？

Many thanks!

[Tom94]

Hi there, color spaces are unfortunately a bit of nightmare when trying to reproduce existing NeRF research, hence all these settings.

• testbed.color_space: this is the color space in which testbed composits the background after rendering. It should always be linear to get correct colors (especially with DoF), except when comparing PSNR numbers to original NeRF, mip-NeRF, NSVF, and other derivatives, because they do non-physical sRGB alpha blending. This setting also applies to SDF & volume rendering.
• testbed.nerf.training.linear_colors: this setting refers to the color space in which the NeRF raymarch takes place. To match physical light transport, this setting should be true, but somewhat frustratingly, the reconstruction appears to always look better when raymarching in sRGB space (corresponding to a setting of false), no matter what we tried. Hence the default: false. We have no good explanation for this and find it vexing to say the least.
• testbed.render(..., linear, ...): default=true: this setting controls whether the returned numpy array is in linear or in sRGB colors. This setting doesn't affect composition/rendering in any way and can be thought of purely as a color-space conversion of the final output. Pick whichever format you prefer. That said, our write_image / read_image helper functions from common.py always expect / return linear colors, so these should be preferred in our ecosystem.

You don't need to worry about setting these to anything other than their default values for optimal results. optimize_exposure should work regardless of setting.

Cheers!

[xmk2222]

Wow, thanks for your rapid reply!

One more question: I noticed that the read_image from common.py assumes that the images are in sRGB colors and converts them to linear colors, so does the testbed (nerf_loader.cu?) assumes the images in sRGB as well? (That is to say training images and validation images should be in the same color space sRGB?)

[Tom94]

Yep, read_image does the "right thing" to produce linear colors, regardless of which image is passed.

Same for nerf_loader.cu, i.e. LDR formats such as PNG/JPG are converted from sRGB RGBA32, whilc HDR formats such as EXR are loaded as-is.

(One more detail: everything in testbed/read_image is using pre-multiplied alpha, which is yet another axis along which color spaces can be weird. You don't need to worry about this, because image formats like PNG specify whether they have pre-multiplied alpha or not. In the case of PNG, it's non-premultiplied and the loaders do the conversion.)