Tensorrt.OnnxParser takes too long to parse the onnx model of two-dimensional input

[zhenhao-huang]

Description

Tensorrt.OnnxParser takes too long to parse the onnx model of two-dimensional input, like [1, 80000].80000 is dynamic.I need to complete the transformation of tensorrt model in a short time, but it takes me several days to parse.

Environment

TensorRT Version: 8.5.3.1
NVIDIA GPU: 3090
NVIDIA Driver Version: 525.60.11
CUDA Version: 11.7
CUDNN Version: 8.5.0
Operating System: ubuntu20
Python Version (if applicable): 3.9.3
PyTorch Version (if applicable): 1.13.1+cu117

[zerollzeng]

Can you share the onnx model here? Thanks!

[zerollzeng]

I'm also trying to find those cases to speed up the parsing. Thanks!

[zhenhao-huang]

Finally, I saved time by modifying the loop operation in the network.Maybe you can try to save the loop operation in the onnx model.This is sample.48000.onnx

[zerollzeng]

Can you share the original model that takes a long time to parse? I want to know if we can optimize the parser instead of letting user to modify the onnx. Thanks!

[zhenhao-huang]

You mean the onnx model with a length of 80000?The 48000 length onnx model also takes a long time to parse.This is 80000.onnx

[zerollzeng]

I'm closing this issue and I will see if we can improve it. Thanks for sharing the model!

[zerollzeng]

Can you also share how this model is generated? Thanks!

[zhenhao-huang]

def forward(self, input):
    one_hot = torch.FloatTensor(np.eye(11)[8].reshape(1, -1)).cuda()
    style_emb = self.obj_vector(one_hot).unsqueeze(1)
    frame_num = input.shape[1] // 320
    hidden_states = self.encoder(input, self.dataset, frame_num=frame_num).last_hidden_state

    vertice_emb = style_emb
    for i in range(frame_num):
        vertice_input = self.PPE(vertice_emb)
        mask1 = self.biased_mask[:, :vertice_input.shape[1], :vertice_input.shape[1]].clone().detach().to(device=self.device)
        mask2 = enc_dec_mask(self.device, self.dataset, vertice_input.shape[1], hidden_states.shape[1])
        vertice_out = self.transformer_decoder(vertice_input, hidden_states, tgt_mask=mask1, memory_mask=mask2)
        new_output = self.vertice_map(vertice_out[:,-1,:]).unsqueeze(1)
        new_output = new_output + style_emb
        vertice_emb = torch.cat((vertice_emb, new_output), 1)
    return vertice_out

This is a structure close to the original network.