S2Convolution requires square inputs

[meder411]

Why does the S2Convolution layer require square inputs? Is there a way around this to explore the use of your convolutions on equirectangular images?

[pochoi]

https://www.cs.dartmouth.edu/~geelong/sphere/

Here is a nice picture on how grid points lying on sphere. The dimension (2 * b_in, 2 * b_in) of the input image is the design of the grid points on the sphere. There are total 2 * b_in * 2 * b_in black dots in the picture.

Such grid point design is the convention of the discrete spherical harmonics transform used in this work.

[meder411]

Ah, okay. So processing an equirectangular projection would require a resampling of the image?

Equirectangular images are usually (180, 360) resolution--essentially unwrapping the sphere around the vertical axis. So I guess I'd need to resample it so that the latitudinal dimension is upsampled by a factor of 2?

[pochoi]

Why not? In the MINST example, the projection of the 2D image to spherical image can be considered as a kind of resampling. The projection/resampling done in the MNIST example is not the only way. We should feel free to try different projection/resampling schemes to see which works better.

[meder411]

Okay, thanks for the explanation! I'll explore a bit.

[weberhen]

I thought that the MNIST dataset is projected on the northern hemisphere only, in which case having a square image makes sense. But once one works on the full sphere, then we need to resample since the spherical projection are 180° - 360° as @meder411 said.