Node [Div] parse error: Different shapes case is not supported with constant inputs: Div in function 'parseMul'

[RAJA-PARIKSHAT]

System information (version)

• OpenCV => 4.5.5-dev
• Operating System / Platform => ubuntu 20.04

Detailed description

I am running a PyTorch model FOMM following the GitHub: https://github.com/AliaksandrSiarohin/first-order-model
I have used the pre-trained weights, the link can be found in the above link.
I want to convert the generator model to ONNX and then run it using OpenCV.
I successfully converted the model into ONNX format but getting error when reading through opencv

[ERROR:0] global /tmp/pip-req-build-th1mncc2/opencv/modules/dnn/src/onnx/onnx_importer.cpp (718) handleNode DNN/ONNX: ERROR during processing node with 2 inputs and 1 outputs: [Div]:(268)
Traceback (most recent call last):
File "opencv_inference.py", line 4, in
net = cv2.dnn.readNetFromONNX("/home/parikshat/Parikshat_Office/codes/generator_model_list.onnx")
cv2.error: OpenCV(4.5.4) /tmp/pip-req-build-th1mncc2/opencv/modules/dnn/src/onnx/onnx_importer.cpp:739: error: (-2:Unspecified error) in function 'handleNode'

Node [Div]:(268) parse error: OpenCV(4.5.4) /tmp/pip-req-build-th1mncc2/opencv/modules/dnn/src/onnx/onnx_importer.cpp:1591: error: (-213:The function/feature is not implemented) Different shapes case is not supported with constant inputs: Div in function 'parseMul'

Steps to reproduce

I modified generator.py file from above github

import torch
from torch import nn

import torch.nn.functional as F
from modules.util import ResBlock2d, SameBlock2d, UpBlock2d, DownBlock2d
from modules.dense_motion import DenseMotionNetwork
from grid_sample import bilinear_grid_sample

class OcclusionAwareGenerator(nn.Module):
   
    def __init__(self, num_channels, num_kp, block_expansion, max_features, num_down_blocks,
                 num_bottleneck_blocks, estimate_occlusion_map=False, dense_motion_params=None, estimate_jacobian=False):
        super(OcclusionAwareGenerator, self).__init__()

        if dense_motion_params is not None:
            self.dense_motion_network = DenseMotionNetwork(num_kp=num_kp, num_channels=num_channels,
                                                           estimate_occlusion_map=estimate_occlusion_map,
                                                           **dense_motion_params)
            self.dense_motion_network
        else:
            self.dense_motion_network = None

        self.first = SameBlock2d(num_channels, block_expansion, kernel_size=(7, 7), padding=(3, 3))

        down_blocks = []
        for i in range(num_down_blocks):
            in_features = min(max_features, block_expansion * (2 ** i))
            out_features = min(max_features, block_expansion * (2 ** (i + 1)))
            down_blocks.append(DownBlock2d(in_features, out_features, kernel_size=(3, 3), padding=(1, 1)))
        self.down_blocks = nn.ModuleList(down_blocks)

        up_blocks = []
        for i in range(num_down_blocks):
            in_features = min(max_features, block_expansion * (2 ** (num_down_blocks - i)))
            out_features = min(max_features, block_expansion * (2 ** (num_down_blocks - i - 1)))
            up_blocks.append(UpBlock2d(in_features, out_features, kernel_size=(3, 3), padding=(1, 1)))
        self.up_blocks = nn.ModuleList(up_blocks)

        self.bottleneck = torch.nn.Sequential()
        in_features = min(max_features, block_expansion * (2 ** num_down_blocks))
        for i in range(num_bottleneck_blocks):
            self.bottleneck.add_module('r' + str(i), ResBlock2d(in_features, kernel_size=(3, 3), padding=(1, 1)))

        self.final = nn.Conv2d(block_expansion, num_channels, kernel_size=(7, 7), padding=(3, 3))
        self.estimate_occlusion_map = estimate_occlusion_map
        self.num_channels = num_channels

    def deform_input(self, inp, deformation):
        _, h_old, w_old, _ = deformation.shape
        _, _, h, w = inp.shape
        if h_old != h or w_old != w:
            deformation = deformation.permute(0, 3, 1, 2)
            deformation = F.interpolate(deformation, size=(h, w), mode='bilinear')
            deformation = deformation.permute(0, 2, 3, 1)
        return bilinear_grid_sample(inp, deformation)

    def forward(self, source_image, kp_driving, kp_source):
        # Encoding (downsampling) part
        
        
        out = self.first(source_image)
        for i in range(len(self.down_blocks)):
            out = self.down_blocks[i](out)

        # Transforming feature representation according to deformation and occlusion
        #output_dict = {}
        output_list = []
        if self.dense_motion_network is not None:
            dense_motion = self.dense_motion_network(source_image=source_image, kp_driving=kp_driving,
                                                     kp_source=kp_source)
            
            #output_dict['mask'] = dense_motion['mask']
            output_list.append(dense_motion[1])
            #output_dict['sparse_deformed'] = dense_motion['sparse_deformed']
            output_list.append(dense_motion[0])
            if len(dense_motion) > 3:
                occlusion_map = dense_motion[3]
                #output_dict['occlusion_map'] = occlusion_map
                output_list.append(occlusion_map)
            else:
                occlusion_map = None
            deformation = dense_motion[2]
            out = self.deform_input(out, deformation)

            if occlusion_map is not None:
                if out.shape[2] != occlusion_map.shape[2] or out.shape[3] != occlusion_map.shape[3]:
                    occlusion_map = F.interpolate(occlusion_map, size=out.shape[2:], mode='bilinear')
                out = out * occlusion_map

            #output_dict["deformed"] = self.deform_input(source_image, deformation)
            output_list.append(self.deform_input(source_image, deformation))
        # Decoding part
        out = self.bottleneck(out)
        for i in range(len(self.up_blocks)):
            out = self.up_blocks[i](out)
        out = self.final(out)
        out = F.sigmoid(out)

        #output_dict["prediction"] = out
        output_list.append(out)
        return output_list

You can see in the forward pass of the generator, it takes three inputs, these are all fixed shapes inputs i.e
source: (1,3,256,256)
kp_driving: (1,10,2)
kp_source: (1,10,2)

and output is also a 5 element list with fixed-sized tensors.
Now when I run the following code in other .py file importing OcclussionAwareGenerator:

generator = OcclusionAwareGenerator(**configuration['model_params']['generator_params'],
                                        **configuration['model_params']['common_params']).to('cpu')
generator_checkpoints = torch.load(PATH_TO_CHECKPOINTS, map_location=  torch.torch.device('cpu'))
generator.load_state_dict(generator_checkpoints)
generator.eval()

source = torch.randn(size = (1,3,256,256), dtype = torch.float32, device= 'cpu')
kp_dummy_source = torch.randn(size = (1,10,2), dtype= torch.float32, device= 'cpu')
kp_dummy_driving = kp_dummy_source.copy()

torch.onnx.export(generator, (source, kp_dummy_driving[0], kp_dummy_source), f = 'generator_model.onnx', opset_version= 11, export_params= True,do_constant_folding= True,
                  input_names= ['source', 'kp_driving', 'kp_source'],output_names= ['mask', 'sparse_deformed', 'occlusion_map', 'deformed', 'prediction'], dynamic_axes= None )

It compiles the model in ONNX format but if I run the following, I get the error

import cv2
net = cv2.dnn.readNetFromONNX("generator_model.onnx")

The model is running in ONNX runtime, basically, I want to run the model on CPU, for that, I want to run OpenCV inference as it will run efficiently.

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues,
• I updated to latest OpenCV version and the issue is still there. compiled cv2 with the master branch and checked the version 4.5.5-dev

[alalek]

OpenCV(4.5.4)

Follow the issue's template. Upgrade first.

[RAJA-PARIKSHAT]

@alalek Now the template is ok? I have OpenCV 4.5.4 installed.

[alalek]

https://github.com/opencv/opencv/blame/4.5.5/.github/ISSUE_TEMPLATE.md#L50

[RAJA-PARIKSHAT]

@alalek I hope now the template is fine. Please help me with the issue.

[alalek]

OpenCV => 4.5.4
cv2.error: OpenCV(4.5.4)
compiled cv2 with the master branch and checked the version 4.5.5-dev

Please align versions.

---

Please attach generator_model.onnx file or similar file (with random weights) for the same problem.

[RAJA-PARIKSHAT]

@alalek
A quick check that I have compiled cv2 with the master branch

import cv2
cv2.version
'4.5.5-dev'

I have also attached a link to the generator_model.onnx file https://drive.google.com/file/d/1epWKi7fX_Dcq2upN4U_FNddpcgf9gO86/view?usp=sharing

[alalek]

You need access

no access to the file.

[RAJA-PARIKSHAT]

@alalek https://drive.google.com/file/d/1epWKi7fX_Dcq2upN4U_FNddpcgf9gO86/view?usp=sharing
now check??
you mean no access to generator_model.onnx file?

[RAJA-PARIKSHAT]

@alalek now the files available to you? or still need access

[asmorkalov]

@RAJA-PARIKSHAT Arithmetic layers have been reworked in OpenCV. Is the issue still relevant for 4.9.0?