TensorRT Example TypeError

[djj0s3]

Describe the bug

Running the example in the repo for TensorRT receiving this error:

 File "/root/.cache/huggingface/modules/diffusers_modules/git/stable_diffusion_tensorrt_txt2img.py", line 706, in to
    super().to(torch_device, silence_dtype_warnings)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/diffusers/pipelines/pipeline_utils.py", line 643, in to
    module.to(torch_device, torch_dtype)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1126, in to
    device, dtype, non_blocking, convert_to_format = torch._C._nn._parse_to(*args, **kwargs)
TypeError: to() received an invalid combination of arguments - got (str, bool), but expected one of:
 * (torch.device device, torch.dtype dtype, bool non_blocking, bool copy, *, torch.memory_format memory_format)
 * (torch.dtype dtype, bool non_blocking, bool copy, *, torch.memory_format memory_format)
 * (Tensor tensor, bool non_blocking, bool copy, *, torch.memory_format memory_format)

Reproduction

Run the example in the repo for TensorRT: https://github.com/huggingface/diffusers/blob/main/examples/community/README.md#tensorrt-text2image-stable-diffusion-pipeline

Logs

No response

System Info

cuda: "11.7"
python_version: "3.9.16"
system_packages:
- "python3-pip"
python_packages:
- "diffusers==0.15.1"
- "torch==2.0.0"
- "ftfy==6.1.1"
- "scipy==1.9.3"
- "transformers==4.25.1"
- "accelerate==0.15.0"
- "huggingface-hub==0.13.2"
- "nvidia-pyindex==1.0.9"
- "tensorrt==8.6.0"

[init-22]

@djj0s3 Because in the at line number 657 in pipeline_utils.py torch_dtype is False, so I've added

if not torch_dtype: torch_dtype = torch.float16

before module.to(torch_device, torch_dtype)

but now I am getting the following error,
Exporting the operator 'aten::scaled_dot_product_attention' to ONNX opset version 17 is not supported.

Anybody have any hint on this? seems like it has to do with the TRT and ONNX versions

@djj0s3 can you still try the changes and see if it works, I changed my TRT, ONNX version, might have got them messed up

[init-22]

Also, in the same (to) function
def to( self, torch_device: Optional[Union[str, torch.device]] = None, torch_dtype: Optional[torch.dtype] = None, silence_dtype_warnings: bool = False, ):

which means torch_dtype should be None if it is not passed but I found that is it is False at line number 657 while debugging, anybody have any idea? I actually wanted to change
torch_dtype: Optional[torch.dtype] = None,

to

torch_dtype: Optional[torch.dtype] = torch.float16

[djj0s3]

No luck, but I sense you're right on the version numbers of the packages. Interestingly looking at examples/community/stable_unclip.py the call doesn't include the boolean:

diffusers/examples/community/stable_unclip.py

Line 180 in 1d7b4b6

def to(self, torch_device: Optional[Union[str, torch.device]] = None):

def to(self, torch_device: Optional[Union[str, torch.device]] = None):
self.decoder_pipe.to(torch_device)
super().to(torch_device)

[init-22]

Yeah interesting!

[djj0s3]

Update and some progress! This example needs a big ol disclaimer. Getting it all running even on a lambdalabs instance was super not fun.

pre-reqs:
You need the latest stable NVIDIA drivers for your GPU. If you're using a cloud instance, your cloud GPU may not be running it. So get that all squared away first. Then some other dependencies:

  cuda: "11.7"
  python_version: "3.9.16"
  system_packages:
    - "python3-pip"
  python_packages:
    - "pip==23.1"
    - "torchvision==0.14.1"
    - "torchaudio==0.13.1"
    - "diffusers==0.15.1"
    - "ftfy==6.1.1"
    - "scipy==1.10.1"
    - "transformers==4.28.1"
    - "accelerate==0.17.1"
    - "huggingface-hub==0.13.4"
    - "tensorrt==8.6.0"
    - "torch==1.13.1" 
    - "nvidia-pyindex==1.0.9"
    - pip install colored polygraphy --extra-index-url=https://pypi.ngc.nvidia.com --trusted-host pypi.ngc.nvidia.com
    - pip install onnx --extra-index-url=https://pypi.ngc.nvidia.com --trusted-host pypi.ngc.nvidia.com
    - pip install onnx_graphsurgeon --extra-index-url=https://pypi.ngc.nvidia.com --trusted-host pypi.ngc.nvidia.com
    - pip install onnxruntime --extra-index-url=https://pypi.ngc.nvidia.com --trusted-host pypi.ngc.nvidia.com

[FluttyProger]

@djj0s3 Because in the at line number 657 in pipeline_utils.py torch_dtype is False, so I've added

if not torch_dtype: torch_dtype = torch.float16

before module.to(torch_device, torch_dtype)

but now I am getting the following error, Exporting the operator 'aten::scaled_dot_product_attention' to ONNX opset version 17 is not supported.

Anybody have any hint on this? seems like it has to do with the TRT and ONNX versions

@djj0s3 can you still try the changes and see if it works, I changed my TRT, ONNX version, might have got them messed up

Hello this help me to fix 'Exporting the operator 'aten::scaled_dot_product_attention' to ONNX opset version 17 is not supported.'

pytorch/pytorch@d06d195

[Metal079]

Hello this help me to fix 'Exporting the operator 'aten::scaled_dot_product_attention' to ONNX opset version 17 is not supported.'

pytorch/pytorch@d06d195

Thanks that helped get passed that it just seems to stop after a bit now

(venv) metal@Pablo-Desktop:/RT_TEST$ cd /home/metal/RT_TEST ; /usr/bin/env /home/metal/RT_TEST/venv/bin/python /home/metal/.vscode-server/extensions/ms-python.python-2023.6.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 39999 -- /home/metal/RT_TEST/test.py
/home/metal/RT_TEST/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py:637: FutureWarning: cached_download is the legacy way to download files from the HF hub, please consider upgrading to hf_hub_download
warnings.warn(
Fetching 16 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:00<00:00, 63550.06it/s]
Running inference on device: cuda:0
Building Engines...
Engine build can take a while to complete
Building Engines...
Engine build can take a while to complete
Building Engines...
Engine build can take a while to complete
Building TensorRT engine for /home/metal/.cache/huggingface/hub/models--stabilityai--stable-diffusion-2-1/snapshots/f7f33030acc57428be85fbec092c37a78231d75a/onnx/clip.opt.onnx: /home/metal/.cache/huggingface/hub/models--stabilityai--stable-diffusion-2-1/snapshots/f7f33030acc57428be85fbec092c37a78231d75a/engine/clip.plan
[W] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage and speed up TensorRT initialization. See "Lazy Loading" section of CUDA documentation https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#lazy-loading
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 681641234
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 681641234
[W] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[I] Configuring with profiles: [Profile().add('input_ids', min=(1, 77), opt=(1, 77), max=(4, 77))]
[I] Loading tactic timing cache from /home/metal/.cache/huggingface/hub/models--stabilityai--stable-diffusion-2-1/snapshots/f7f33030acc57428be85fbec092c37a78231d75a/timing_cache
[W] Timing cache file /home/metal/.cache/huggingface/hub/models--stabilityai--stable-diffusion-2-1/snapshots/f7f33030acc57428be85fbec092c37a78231d75a/timing_cache not found, falling back to empty timing cache.
[I] Building engine with configuration:
Flags | [FP16]
Engine Capability | EngineCapability.DEFAULT
Memory Pools | [WORKSPACE: 24563.38 MiB, TACTIC_DRAM: 24563.38 MiB]
Tactic Sources | []
Profiling Verbosity | ProfilingVerbosity.DETAILED
Preview Features | [DISABLE_EXTERNAL_TACTIC_SOURCES_FOR_CORE_0805]
[W] kFASTER_DYNAMIC_SHAPES_0805 preview feature is disabled.
(venv) metal@Pablo-Desktop:~/RT_TEST$

[init-22]

@FluttyProger Thanks for the update, @Metal079 this conversion should roughly take ~30 mins as per the documentation,
not sure what to do next to resolve this! Lets look at the change log of pytorch and onnx to see if we find something there

[FluttyProger]

Yeah, 30 mins is a lame, but fortunately we have to do it once, then loading the tensorrt engine from cache takes seconds

[sayakpaul]

Hi @djj0s3 could you maybe open a PR to include your findings? This would help us a ton.

[sayakpaul]

Community pipelines are not officially maintained by us. Pinging the original author: @asfiyab-nvidia.

[abhijitpal1247]

@djj0s3 what's the minimum GPU requirement is for using this TensoRT community example? And were you able to get any speed-up during inference?

[asfiyab-nvidia]

Hi, addressing a few concerns:

1. The fix for TypeError was included in a follow up PR.
2. This pipeline works with PyTorch < 2.0. The Exporting the operator 'aten::scaled_dot_product_attention' to ONNX opset version 17 is not supported. comes from using a higher torch version
3. We will be updating the dependency list in a PR in the next few weeks