TF-TRT Improve matrix multiplication conversion and enable dynamic shape mode

[tfeher]

This PR improves the MatMul and BatchMatMul converters.

• Unnecessary transpose of weights are removed. Transposing weights for IMatrixMultiplyLayer is not necessary, because IMatrixMultiplyLayer can directly pass the transpose flags to the underlaying GEMM call, which can use it to access elements with the correct stride without any actual transposition.
• Restrictions caused by previous weight transpose ops eliminated.
• Enabled explicit batch and dynamic shape input.
• IFullyConnectedLayer (FC) usage fixed:
  • FC layer is preferred over IMatrixMultiply because it is expected to give better performance. Moreover, currently only FC layer supprorts INT8 precision.
  • Fixed and relaxed FC layer conversion condition.
  • Fixed input tensor_a shaped handling. In dynamic shape mode care has to be taken to retain static dim where available and not to confuse unknown dims with -1 wildcard.
  • Enabled rank > 2 for weights.
  • Fixed conversion of BatchMatMul to FC: broadcast now preserves the information whether the input is tensor or weight, so that we can correctly check FC compatibility condition.

BatchMatMul involves a potential broadcast step. TRT requires that the input tensors have the same rank, with 1 values filled in the dimensions which need to be broadcasted. A helper function BroadcastTensors was added to make the tensors match in rank. In dynamic shape mode we need shape inference for this step. The DynamicReshape function was modified to allow insertion of multiple singleton dimensions.

Tagging @bixia1 for review and @DEKHTIARJonathan for visibility.
Tracker: #45481

[bixia1]

Thanks for your work!
I finished reviewing the code, but not the test yet. Sending out the comments I have now.

[tfeher]

Thanks @bixia1 for the review, I have addressed the issues!

[bixia1]

Please remember to rebase and squash.

[tfeher]

Thanks @bixia1 for the additional comments, I have addressed the issues!

[bixia1]

Thanks for your work! Just one remaining thing: there is one place where you said you put a more detail description but I couldn't find it by searching the string in the code.

[tfeher]

I have fixed the missing comment.