🐛 [Bug] Tensor RT detecting input shapes incorrectly

[srelbo]

Bug Description

Compiling Torch traced model to TensorRT fails with tensor concat failure. Tensor RT detecting input shapes incorrectly

To Reproduce

Steps to reproduce the behavior:

1. JIT Trace our custom model -> Works
2. Run sample inputs through traced model -> Works
3. Try to convert to TensorRT -> Fails with input shape being detected incorrectly

ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::dispatchComputeOutputExtents::1147] Error Code 4: Miscellaneous ((Unnamed Layer* 62) [Concatenation]: error while computing output extent)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::dispatchComputeOutputExtents::1147] Error Code 4: Miscellaneous ((Unnamed Layer* 62) [Concatenation]: error while computing output extent)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::dispatchComputeOutputExtents::1147] Error Code 4: Miscellaneous ((Unnamed Layer* 62) [Concatenation]: error while computing output extent)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::dispatchComputeOutputExtents::1147] Error Code 4: Miscellaneous ((Unnamed Layer* 62) [Concatenation]: error while computing output extent)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::dispatchComputeOutputExtents::1147] Error Code 4: Miscellaneous ((Unnamed Layer* 62) [Concatenation]: error while computing output extent)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::dispatchComputeOutputExtents::1147] Error Code 4: Miscellaneous ((Unnamed Layer* 62) [Concatenation]: error while computing output extent)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::dispatchComputeOutputExtents::1147] Error Code 4: Miscellaneous ((Unnamed Layer* 62) [Concatenation]: error while computing output extent)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::dispatchComputeOutputExtents::1147] Error Code 4: Miscellaneous ((Unnamed Layer* 62) [Concatenation]: error while computing output extent)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::dispatchComputeOutputExtents::1147] Error Code 4: Miscellaneous ((Unnamed Layer* 62) [Concatenation]: error while computing output extent)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::dispatchComputeOutputExtents::1147] Error Code 4: Miscellaneous ((Unnamed Layer* 62) [Concatenation]: error while computing output extent)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::dispatchComputeOutputExtents::1147] Error Code 4: Miscellaneous ((Unnamed Layer* 62) [Concatenation]: error while computing output extent)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::dispatchComputeOutputExtents::1147] Error Code 4: Miscellaneous ((Unnamed Layer* 62) [Concatenation]: error while computing output extent)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [concatenationLayer.cpp::estimateOutputDims::65] Error Code 4: Internal Error ((Unnamed Layer* 62) [Concatenation]: all concat input tensors must have the same number of dimensions. Input 0 shape: [1,10,5,5,1]. Input 1 shape: [1,10,3,5].)

Expected behavior

TensorRT is detecting the input shape incorrectly . Instead of Input 0 shape: [1,10,5,5,1]. Input 1 shape: [1,10,3,5].) it should be [1, 10, 5, 5, 1] it should be [1, 10, 5, 5]

Environment

• Torch-TensorRT Version (e.g. 1.0.0): 1.3.0
• PyTorch Version (e.g. 1.0):1.13.1
• CPU Architecture: Intel
• OS (e.g., Linux): Ubuntu
• How you installed PyTorch (conda, pip, libtorch, source): pip3
• Build command you used (if compiling from source): NA
• Are you using local sources or building from archives: NA
• Python version: 3.10
• CUDA version:11.7
• GPU models and configuration: 3090
• Any other relevant information:
  The same traced (and also scripted) model converts fine to Pytorch Lite. But conversion to tensor RT fails.

Additional context

        # Try tensor RT
        print("Trying to convert to TensorRT")
        _traced_model = try_tracing_model(_nn, _source_image, _driving_image[0:1], fps=_fps)
        print("Trace complete...")

        enabled_precisions = {torch.half}  # Run with fp16
        with torch_tensorrt.logging.debug():
            trt_ts_module = torch_tensorrt.compile(
                _traced_model.to('cuda'),
                inputs=(_source_image.to('cuda'), _driving_image[0:1].to('cuda')),
                enabled_precisions=enabled_precisions,
                truncate_long_and_double=True,
                require_full_compilation=False
            )

            print("Converted to TensorRT")
        result = trt_ts_module((_source_image, _driving_image[0:1]))

Any help would be much appreciated! Thank you!

[peri044]

@srelbo In your debug logs, there should be a line which says "Adding Input ........ " which shows the input shape and dtype.
Can you see if you're seeing the wrong shape [1, 10, 3, 5] in the logs ?

[srelbo]

Thanks for taking a look @peri044 ! Here are the full logs -- https://gist.github.com/srelbo/49083069befd1e43f0de9358ca5dcad4

This is where I see the wrong shape --

DEBUG: [Torch-TensorRT] - ITensor name: (Unnamed Layer* 58) [Concatenation]_output
DEBUG: [Torch-TensorRT] - ITensor shape: [1, 10, 5, 3]
DEBUG: [Torch-TensorRT] - ITensor type: Float32
DEBUG: [Torch-TensorRT] - Shuffle to: [0, 1, 3, 2]
DEBUG: [Torch-TensorRT] - Output tensor shape: [1, 10, 3, 5]

[github-actions]

This issue has not seen activity for 90 days, Remove stale label or comment or this will be closed in 10 days