Question about the radius setting.

[YuDeng]

Hi, a wonderful work!

I am wondering why the radius of the cone is set to r=2/sqrt(12)*pixel_size.

I know that this setting is to ensure that the variance of the cone matches that of the pixel in world coordinate space. I'm just curious about the derivation. Could you please give me some hint on how to get to this result?

Thanks,
Yu

[jonbarron]

The derivation follows from the fact that the variance of a uniform distribution over [0, 1] is 1/12: https://en.wikipedia.org/wiki/Continuous_uniform_distribution#Moments. So if you want a standard deviation, you take the square root of variance, and then scale it by the pixel size. The factor of 2 is to turn a diameter into a radius, I think, though I'm not sure.

[YuDeng]

Thanks for the reply.

I can get 1/sqrt(12) for the std of x or y. What I don't quite understand here is that why to multiply 2 with the std. If we want to turn diameter into radius, shouldn't we divide std by 2 so that the final result turns to be 1/(2*sqrt(12)) ?

[jonbarron]

yeah, that sounds right. I remember having the same thought and then verifying that this was correct, albeit counter-intuitive. I might have messed up the math though! That part of the code doesn't have unit test coverage.

[kwea123]

Btw, why is here -2?

mipnerf/internal/datasets.py

Line 196 in ee5cbb8

dx = np.concatenate([dx, dx[:, -2:-1, :]], 1)

dx is the difference to its neighbor, so for the rightmost pixel it should be simply dx[:, -1:], isn't it?
For example, let's assume we have an image of width 3, with the above code it means the radius of the rightmost pixel is the direction difference between pixel 0 and pixel 1, which is confusing to me.

[kwea123]

@jonbarron @YuDeng

The derivation follows from the fact that the variance of a uniform distribution over [0, 1] is 1/12

That's correct, but I think that it's only valid if you generate rays randomly and uniformly inside pixels. However, in your ray generation

mipnerf/internal/datasets.py

Lines 183 to 186 in ee5cbb8

	camera_dirs = np.stack(
	[(x - self.w * 0.5 + 0.5) / self.focal,
	-(y - self.h * 0.5 + 0.5) / self.focal, -np.ones_like(x)],
	axis=-1)

The rays are always fixed to be passing through pixel centers, so I don't know why uniform distribution comes into play here.

From the description, r is the radius of a pixel in world coordinate (same as camera coordinate actually), so it should be as simple as dx/2 here, instead of 2*dx/sqrt(12):

mipnerf/internal/datasets.py

Line 200 in ee5cbb8

radii = dx[..., None] * 2 / np.sqrt(12)

I've tried changing it to dx/(2*sqrt(12)) as the discussion above, but found it to produce inferior quality. I wonder if you have done any experiments on this r, otherwise I think dx/2 should be the correct way... and numerically 1/2 is close to 2/sqrt(12), so maybe that's why your 2/sqrt(12) also works? But that value is not based on any valid explanation.

Hope to get some more clarification, thanks.

[jonbarron]

I think you might be right. I also re-derived this math a while back and got the answer you got, and I also tried it out and saw that it hurt performance, so I stuck with the math that we have here. It's entirely possible that there's a bug in the math/code here, which is definitely worth exploring. It's also possible that there is some advantage of using a radius that is slightly bigger or smaller than the true radius. For example, if you used a wider radius, you'd basically be having NeRF use a "camera" or "sensor" whose PSF has been blurred slightly, which maybe is a good thing in some circumstances? Maybe the best thing to do here is to introduce some radius_multiplier=~4 hyperparameter, and set the code to use radii = radius_multiplier * dx / (2*sqrt(12)). This would be fix this bug, but not degrade performance. And you could imagine tuning this hyperparameter depending on what assumptions you want to make about the optical prefilter that your data is using. I don't find this answer particularly satisfying but it's the best I can come up with. This definitely merits further investigation from my end (which may or may not happen) and I'll let you know if I figure something out.

[bsuleymanov]

@kwea123 @jonbarron I had another explanation why 2/sqrt(12) worked, but the derivation was different.

We have a square (in case where we use only dx and ignore dy) pixel with width dx. Area of that pixel is dx^2. When we trace a cone we want it basis to be the circle with area approximately equal to the pixel's area. Area of the cone's basis is pi * r^2. pi * r^2 = dx^2 => r = dx / sqrt(pi) ~ dx / 0.56. 2 * dx / sqrt(12) ~ 0.577.

[jonbarron]

Oh cool, I think this is what I did! Sorry this is so complicated, I didn't do a good job of recording my notes when working through the math for this paper.

[jonbarron]

Yeah I remember first using dx as the radius, which gives you a circle that is inscribed inside of the pixel, and then thinking "no, it should be bigger than that" and I must have tried to match the area or do something similar. I think what I ended up doing was taking the mean or geometric mean of the radius you get when you inscribe the circle and circumscribe the circle.

[longbowzhang]

Hi all,
I just would like to make sure that the setting of radius has nothing to do with neither the variance of the pixel’s footprint as mentioned in the paper nor a normal distribution, right?

[jonbarron]

I'm not sure I understand the question. The radius of a pixel is a direct function of the size of the pixel's footprint on the image plane. Then the radius is used to construct a normal distribution.

[longbowzhang]

Hi @jonbarron,

[1] > Hi all, I just would like to make sure that the setting of radius has nothing to do with neither the variance of the pixel’s footprint as mentioned in the paper nor a normal distribution, right?

normal distribution should be corrected to uniform distribution (sorry for my mistake).
Based on your answer #5 (comment), the radius is derived mainly from a geometric perspective. Then its value has nothing to do with a distribution/variance. Right?

[2] Besides, I still do not know where the sqrt(12) comes from. I checked the arithmetic mean and the geometric mean of the radius of the inscribed circle and circumscribed circle, and found out that neither gives a sqrt(12). (Hope I do not get the math wrong again)

[3] Furthermore, I am wondering this radius setting is constant for all the pixels across the image plane, even for pixels near the border where the intersection of the cone and the image plane is an ellipsoid instead of a circle?

[jonbarron]

sqrt(12) is just the standard deviation of a uniform distribution defined on [0, 1]. Here I'm deriving it using the definition of variance (E[x^2] - E[x]^2): https://www.wolframalpha.com/input?i=Sqrt%5BIntegrate%5Bx%5E2%2C+%7Bx%2C+0%2C+1%7D%5D+-+Integrate%5Bx%2C+%7Bx%2C+0%2C+1%7D%5D%5E2%5D

[longbowzhang]

I finally figure it out that the variance of uniform distribution within the circle of radius d is (d^2)/4 https://www.wolframalpha.com/input?i=Integrate%5B1%2F%28Pi*d%5E2%29*r*%28r*sin%28theta%29%29%5E2%2C+%7Br%2C+0%2C+d%7D%2C+%7Btheta%2C+0%2C+2*Pi%7D%5D. By setting it equal to 1/12, we can easily derive that d = 2/sqrt(12).

[qhdqhd]

Is dx all the same for different pixel rays?

`

Distance from each unit-norm direction vector to its x-axis neighbor.

dx = [
    np.sqrt(np.sum((v[:-1, :, :] - v[1:, :, :])**2, -1)) for v in directions
]

dx = [np.concatenate([v, v[-2:-1, :]], 0) for v in dx]

# Cut the distance in half, and then round it out so that it's
# halfway between inscribed by / circumscribed about the pixel.

radii = [v[..., None] * 2 / np.sqrt(12) for v in dx]`

[qhdqhd]

@longbowzhang
for [3]:
I think dx is all the same for different pixel rays. 这个半径设置对于图像平面上的所有像素都是恒定的.

[longbowzhang]

@qhdqhd it seems like the authors set that value constant for all the pixels even though mathematically I personally think dx should be adaptive for every pixel.