R2Conv acts on 5-D tensor

[yokinglou]

There is a situation when images are stacked twice. The shape of the structured image tensor is (B, S, C, H, W) where B represents batch size and S the number of stacked images.
Could R2conv directly act or broadcast on such a 5-D tensor?

[Gabri95]

Hi @yokinglou

Let me understand exactly what you need to do.
I see three different cases.

1. the S images need to be processed independently using the same weights. Then you can just reshape to (B*S, C, H, W) pushing them in the batch dimension

2. the S images are just additional features which can and need to be processed together. The you can reshape the input to (B, S*C, H, W). In our library, you need to define the FieldType as S times larger. Suppose that an image (one of the S images) has FieldType

gs = e2cnn.gspaces.Rot2dOnR2(8)
ft = e2cnn.nn.FieldType(gs, [gs.regular_repr]*3)

then your (reshaped) input can have field type

ft_S = e2cnn.nn.FieldType(gs, [gs.regular_repr]*3 * S)

Then, you can use the R2Conv as usual.

3. the S images are additional features but they need to be processed independently, i.e. they are processed by their own set of filters and their outputs are independent. You can do this using grouped convolution.
   You do this by using R2Conv and the FieldType like above in 2). However, when you build R2Conv, you need to set the parameter groups=S.

Does this answer your question?
If not, could you provide how you would implement the operation you would like in PyTorch?
I can then help you implement it in our library

Best,
Gabriele

[yokinglou]

Hi Gabriele

Thanks for your detailed reply! I think the first case is what I want to do.
I have tried to reshape the tensor to (B*S, C, H, W) and send it to the network to get the invariant feature (B*S, C'). Reshape the feature back to (B, S, C') at last.
Here is a problem. The occupied memory on the GPU is too large when I increase the S. One way is reducing the batch size to release the occupation while this is not expected.
Is there any way to improve memory efficiency without losing the information included in the original input?

Thanks,
Yoking

[Gabri95]

Hi @yokinglou

I think this is an expected behaviour, as you are effectively increasing the batch size.
What is the meaning of the S images?
From what I have understood, these are just additional images in the batch, right?

I feel like this issues is not really related to our library, but rather to some design choice in your experiments.
Have I misunderstood something?

Unfortunately, I am not sure I understand what you mean in your last question.

Best,
Gabriele

[yokinglou]

I agree with you. I will try more to improve efficiency.
Anyway, thank you for your well-organized code and detailed reply.
This project helps me a lot.

Best,
Yoking

[Gabri95]

Hi @yokinglou ,

I am happy to read this! Thanks!

Please, let me know if you encounter any other issues :)

Best,
Gabriele