Issue with running inverse_canonicalization

[olayasturias]

Hello all,

Thanks for sharing such amazing work! I'm very interested in using and implementing your code, particularly for equivariant tasks.
For learning to use your library, I've been playing around with your notebook under tutorials/images/understanding_discrete_canonicalization.ipynb. The notebook is very straightforward and easy to understand, so congrats for that :)

As a step further, I tried to add the function invert_canonicalization to play with it as follows:

for image in train_loader_tqdm:
  image = image.to(device)
  canonicalized_image = canonicalizer.canonicalize(image)
  outputs = canonicalizer.invert_canonicalization(canonicalized_image)
  [...]

When doing this, some problems and questions were raised:

• First of all, the function was throwing a KeyError because it was searching for a key element named "group" in the canonicalization_info_dict dictionary. However, this key is called "group_element" and not group. I've fixed this, and I will send you the pull request soon :)
• Secondly, I have a question regarding the function get_action_on_image_features under /images/utils.py:
  • Why does the dimension feature_map. shape [1] need to be multiple of num_group? In this example, these dimensions are 3 and 4, respectively. However, with that requirement, the image's channel size would have to be 4 or 8, for example. As far as I understand, this function reverses the canonicalization, that is, rotates the image by a certain angle. So why is the expected dimension different than that of the image? Have I misunderstood what this function is aiming to do?
  • What would happen if the output of the prediction_network (not used in this particular example but exemplified in your README) has a different shape than the input required by the prediction_network? For example, the input is an image, and the output is a vector. In that case, we'd like to apply the rotation action to the vector instead of an input image. Do you store the canonicalization (or inverse canonicalization) transform somewhere so that it can be applied to whichever data shape? I've seen that you have a list of angles under group_element_dict["group_element"]["rotation"]; however, I don't quite understand what these angles represent. Shouldn't it be just one angle?

Thank you so much :)

Olaya

[sibasmarak]

Thank you for the nice feedback and the PR, @olayasturias! I will look at the PR later and merge it.

The get_action_on_image_features applies a group action to a "feature map" depending on the induced representation type. I suppose you are referring to this line where induced_rep_type="regular". We must ensure this for the regular representation type because we need to reshape the feature map later and perform shifts along the group dimension.

However, in your case, when you want to apply the group action to the image, we are dealing with the "scalar" representation type. Therefore, we do not need to check if feature_map.shape[1] needs to be multiple of num_group, and a roto-reflection on the image suffices.

For more details on understanding induced representations, I recommend Section 2 of the Steerable CNNs paper (and, for further reading, Section 2 of General E(2)—Equivariant Steerable CNNs).

Please let me know if the answer needs to be further clarified (or if I have misunderstood your question), otherwise feel free to close this issue.

[olayasturias]

Thank you for your fast and helpul reply. I understand now, thanks :)
This has made me find another key naming error under the scalar induced_rep_type, I have updated the pull request.