How did you get the pose

[WenjiaWang0312]

Hi there. I read in your paper that you can "learn a 3D-aware generative model from unposed 2D images." But in section 3.2.1 you also mentioned that "We sample the camera pose ξ = [R|t] from a pose distribution p ξ ." So I am wondering what's the pose in fact. How to get pose from the unposed images?

[TDeVries]

In GRAF I think it is assumed that the pose distribution is known (i.e. we know the range of possible camera locations on a viewing sphere around the target object), but the poses of individual images in the dataset are not known.

If you have an unposed image and want to find the pose using a trained GRAF model, you could probably do that by optimizing the latent code and camera pose parameters simultaneously such that the model generates an image matching the target.

[WenjiaWang0312]

In GRAF I think it is assumed that the pose distribution is known (i.e. we know the range of possible camera locations on a viewing sphere around the target object), but the poses of individual images in the dataset are not known.

If you have an unposed image and want to find the pose using a trained GRAF model, you could probably do that by optimizing the latent code and camera pose parameters simultaneously such that the model generates an image matching the target.

In the paper they used the dataset VOX and CatFace. I think these images do not have corresponding camera poses? Maybe I should read the code carefully.

[katjaschwarz]

Yes, we assume the rough distribution over camera poses is known (but do not require posed images). You can find the choices for the pose distributions of all datasets in Table 4 of our supplementary material .

[WenjiaWang0312]

Yes, we assume the rough distribution over camera poses is known (but do not require posed images). You can find the choices for the pose distributions of all datasets in Table 4 of our supplementary material .

Thanks.
So you mean that we do not need to provide each image in VOXCeleb with a camera poses? Just a range?

[katjaschwarz]

Yes, exactly. In the code we do this by setting a range for azimuth and polar angle in the config file via parameters u (umin, umax) and v (vmin, vmax):

u = azimuth / 360
v = 0.5* (1-cos(polar * pi/180))

We use u and v here because sampling u, v uniformly in their respective range leads to uniformly distributed poses on the sphere in the given range. Note that sampling azimuth and polar angle directly from a uniform distribution would lead to a pose distribution that is not uniformly distributed on the sphere in the given range.

[Darcy-vision]

Yes, exactly. In the code we do this by setting a range for azimuth and polar angle in the config file via parameters u (umin, umax) and v (vmin, vmax):

u = azimuth / 360 v = 0.5* (1-cos(polar * pi/180))

We use u and v here because sampling u, v uniformly in their respective range leads to uniformly distributed poses on the sphere in the given range. Note that sampling azimuth and polar angle directly from a uniform distribution would lead to a pose distribution that is not uniformly distributed on the sphere in the given range.

Hi, I want to know why the input is randomly sampled pose during training, but after training the Generator can learn the patterns of image changes corresponding to pose changes ? Because the discriminator also has no GT pose to guide the learning. @katjaschwarz

[katjaschwarz]

The image changes correspond to pose changes because the neural radiance field is projected to an image using volume rendering. Conditioning the discriminator is not needed but can help and is used in some later works.

[Darcy-vision]

I understand now. Thank you for your explanation.