Add parameter for Deformable Convolution offset group scalar value

[joeyballentine]

🚀 Feature

Currently, the scalar used to calculate the number of deformable groups is hardcoded at 2. I would like for a parameter to be added that allows this number to be anything in order to have compatibility with repositories such as EDVR which use 3 for this value.

I have already added it myself and was going to submit a PR before reading that I should submit an issue first.

Motivation

I am currently trying to replace the MMdetection Deformable Convolution v2 with the Torchvision one for the EDVR repository. However, for its offsets, it calculates the out_nc size using this formula: self.deformable_groups * 3 * self.kernel_size[0] * self.kernel_size[1]. The usual formula, which the current Torchvision implementation expects, is self.deformable_groups * 2 * self.kernel_size[0] * self.kernel_size[1]. As you can see, they use a 3 in this calculation instead of a 2. I'm not entirely sure why, but it doesn't work unless it uses 3.

This causes an issue when using the Torchvision implementation, as in order to calculate the number of offset groups (called deformable groups in the formula above), it requires that scalar value to be 2.

EDVR Formula

Torchvision Formula

Pitch

I would like for a parameter to be added that would allow me to change this value, like so.

Alternatives

Another alternative could be to allow the number of offset groups to be passed in instead of being auto-calculated, as that is what the MMDetection version does.

Additional context

None.

[datumbox]

@joeyballentine thanks for the detailed write up.

@fmassa Any thoughts on the differences in the two formulas and on whether it's worth making it parameterizable?

[joeyballentine]

After thinking about it more, I think it may be more reasonable to pass in the number of offset groups as an optional parameter instead of the scalar value. If it is not included it gets calculated the way it is now automatically, but if it is included then it alleviates this issue. Thoughts?

[joeyballentine]

Upon further inspection, I now realize I can actually call torch.ops.torchvision.deform_conv2d directly in order to pass in the correct values. While I still think the option of passing the number of offset groups in would be nice, it is good to know that it actually is not necessary to get the functionality I need.

[fmassa]

Hi,

I'm not sure I understand correctly. We do allow to specify the operator to have an arbitrary number of offset groups, by just padding a properly shaped offset and mask tensors.
The 2 which is hard-coded in

vision/torchvision/ops/deform_conv.py

Line 82 in 0985533

n_offset_grps = offset.shape[1] // (2 * weights_h * weights_w)

stands for x, y, which from my understanding is always present.

Also note that in the torchvision case, the line just above infers the number of offset groups from the size of the passed in offset tensor.

[joeyballentine]

@fmassa The issue I was having was due to offsets and masks that were not properly shaped. I don't know why exactly since it's not my code, but the offset size was being created using a different value, which made the auto-calculated offsets groups incorrect.

The solution I found works better anyway, as the original implementation allowed the user to specify any number of offset groups in a configuration file, so allowing that to still happen means that my torchvision version is 1:1 with the original.

[fmassa]

@joeyballentine ok, so from my understanding then this was a user error as the shapes of offsets and masks were not correct, so we couldn't properly infer the number of offset groups.

BTW, I would not recommend calling directly through the torch.ops.torchvision.deform_conv2d as it is an implementation detail and can change at any time without notice. So it might be preferable to fix the code upstream then on relying on internal implementations

[joeyballentine]

@fmassa if you look at the implementation of Deformable convolution that was used to make the mmdetection one, you can see in their ModulatedDeformConvPack they use the same value of 3 https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch/blob/7e69911acfca4183172b829c50b174af721310e8/modules/modulated_dcn.py#L56

Maybe this really is an implementation error on their part, but I feel like if it was, then the EDVR/BasicSR repo would have encounter an issue with this implementation and changed it to use the value of 2 instead. Yet, it still uses 3. Maybe there is something here that could help explain why this value is set to 3 here.

I'll reopen this issue as it seems like this is not solved after all.

[NicolasHug]

I agree with @fmassa that the 2 in torchvision's implementation refers to the h and w dimensions.

From section 3.2 of the DeformConv v2 paper https://arxiv.org/abs/1811.11168:

The output is of 3 K channels, where the first 2K channels correspond to the learned offsets ∆pk, and the remaining K channels are further fed to a sigmoid layer to obtain the modulation scalars ∆mk.

where K is self.kernel_size[0] * self.kernel_size[1]. So the difference between 2 and 3 seems to come from the modulation scalars.

I could be wrong as I'm not super familiar with the paper nor the implementation, but I believe those modulation scalars actually correspond to the mask parameter.

I'll close the issue, please feel free to re-open if there are still some doubts.