Regarding the ray color calculation from the MLP output.

[DinushkaDDS]

Hi,

I was trying to rewrite the NeRF from scratch to understand the technique better and during the implementation I encountered a concern which I need a help. I am new to computer graphics area so there may be a misunderstanding from my end. Please let me know if that's the case.

If I understood the paper correctly, In NeRF we are supposed to calculate the pixel colors of an image by integrating over the ray sample point colors (c) and volume density values (σ) given by the MLP using the provided formula in the paper.

Since we cant integrate, we use below formula to calculate the estimated ray color.

In the provided implementation of this repo, we first calculate the difference between sampled points (δ).
Then we calculate the a value called alpha = 1.0 - tf.exp(-act_fn(raw) * dists) which I think is equivalent to 1 − exp(−σiδi) part of above formula.

Then comes my problem.

weights = alpha * tf.math.cumprod(1.-alpha + 1e-10, axis=-1, exclusive=True)
rgb_map = tf.reduce_sum(weights[..., None] * rgb, axis=-2) # [N_rays, 3]

As per my understanding weights variable is equal to the Ti(1 − exp(−σiδi)) part of the formula (Not Exactly). But shouldn't it be implemented roughly like below according to the paper?

alpha = 1 - np.exp(-volume_density * delta)
T_i = np.exp(-np.cumsum(volume_density * delta))
rgb_map = np.sum(T_i*alpha*colors, axis=1)

why this difference exist? Is my understanding is wrong or am I missing something?

[cx-tian]

I also have the same question, hope someone can explain this.

[DinushkaDDS]

I tried out the NeRF training based on what I understood from the paper. Its just 3 line change compared to the provided implementation.

In above code transmittance is equivalent to Ti term in the equation and alpha equal to (1 − exp(−σiδi)) term. weights is equal to the Ti(1 − exp(−σiδi)) term.
(I added the epsilon value as well since it was included in original implementation to avoid total zero values after multiplication operation and not entirely sure about the depth map calculation)

Nevertheless it seems NeRF model converge regardless of the this difference. But based on my limited testing converging seems to be bit unstable in summation based implementation. Again this need to be tested properly. I just ran the code few times to see everything is working.

Other than that I could not find any details regarding this. If found an explanation please let me know.

[resindraburiza]

I tried to expand the math a little bit and ended up here. I think this is the reason why cumprod is used.

[DinushkaDDS]

Thank you very much!
Should have played with the formulas rather than messing around with the code continuously.

Closing the issue since the question got answered.