Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tensorrt error #19

Open
bryanhpchiang opened this issue Dec 22, 2023 · 17 comments
Open

tensorrt error #19

bryanhpchiang opened this issue Dec 22, 2023 · 17 comments

Comments

@bryanhpchiang
Copy link

TypeError: streamdiffusion.acceleration.tensorrt.compile_unet() got multiple values for keyword argument 'opt_batch_size'

?
@bryanhpchiang
Copy link
Author

image

This part of the code in particular needs some work I think

@bryanhpchiang
Copy link
Author

bryanhpchiang commented Dec 22, 2023 •
edited
Loading

image

@bryanhpchiang
Copy link
Author

These errors are just the result of
image

@olegchomp
Copy link 

from streamdiffusion.acceleration.tensorrt import accelerate_with_tensorrt

stream = accelerate_with_tensorrt(
    stream, "engines", max_batch_size=2,
)

Yep, after changing to tensorrt got an error. But examples scrips work.

@lamquangtuong
Copy link

lamquangtuong commented Dec 25, 2023 •
edited
Loading

self.trt_unet_batch_size is a tuple

StreamDiffusion/src/streamdiffusion/pipeline.py

Line 51 in 62596fd

self.trt_unet_batch_size = (

@ngc-shj
Copy link

ngc-shj commented Dec 26, 2023

I also encountered the same issue.
I think this issue can be solved by specifying the engine_path option.

diff --git a/src/streamdiffusion/acceleration/tensorrt/__init__.py b/src/streamdiffusion/acceleration/tensorrt/__init__.py
index e629567..90a6a07 100644
--- a/src/streamdiffusion/acceleration/tensorrt/__init__.py
+++ b/src/streamdiffusion/acceleration/tensorrt/__init__.py
@@ -139,6 +139,7 @@ def accelerate_with_tensorrt(
             unet_model,
             create_onnx_path("unet", onnx_dir, opt=False),
             create_onnx_path("unet", onnx_dir, opt=True),
+            engine_path=unet_engine_path,
             opt_batch_size=max_batch_size,
             **engine_build_options,
         )
@@ -151,6 +152,7 @@ def accelerate_with_tensorrt(
             vae_decoder_model,
             create_onnx_path("vae_decoder", onnx_dir, opt=False),
             create_onnx_path("vae_decoder", onnx_dir, opt=True),
+            engine_path=vae_decoder_engine_path,
             opt_batch_size=max_batch_size,
             **engine_build_options,
         )
@@ -162,6 +164,7 @@ def accelerate_with_tensorrt(
             vae_encoder_model,
             create_onnx_path("vae_encoder", onnx_dir, opt=False),
             create_onnx_path("vae_encoder", onnx_dir, opt=True),
+            engine_path=vae_encoder_engine_path,
             opt_batch_size=max_batch_size,
             **engine_build_options,
         )`

@8600862
Copy link

8600862 commented Dec 27, 2023

got multiple values for keyword argument 'opt_batch_size' means give 'opt_batch_size' more than once, opt_batch_size=max_batch_size and engine_build_options, just del one of them

@olegchomp
Copy link

olegchomp commented Dec 27, 2023 •
edited
Loading

got multiple values for keyword argument 'opt_batch_size' means give 'opt_batch_size' more than once, opt_batch_size=max_batch_size and engine_build_options, just del one of them

Sorry but i can you explain solution. I've tried with default example from readme and still get this error #61

@8600862
Copy link

8600862 commented Dec 27, 2023

got multiple values for keyword argument 'opt_batch_size' means give 'opt_batch_size' more than once, opt_batch_size=max_batch_size and engine_build_options, just del one of them

Sorry but i can you explain solution. I've tried with default example from readme and still get this error #61

did you comments this

opt_batch_size=max_batch_size,

@8600862
Copy link

8600862 commented Dec 27, 2023

I also encountered the same issue. I think this issue can be solved by specifying the engine_path option.

diff --git a/src/streamdiffusion/acceleration/tensorrt/__init__.py b/src/streamdiffusion/acceleration/tensorrt/__init__.py
index e629567..90a6a07 100644
--- a/src/streamdiffusion/acceleration/tensorrt/__init__.py
+++ b/src/streamdiffusion/acceleration/tensorrt/__init__.py
@@ -139,6 +139,7 @@ def accelerate_with_tensorrt(
             unet_model,
             create_onnx_path("unet", onnx_dir, opt=False),
             create_onnx_path("unet", onnx_dir, opt=True),
+            engine_path=unet_engine_path,
             opt_batch_size=max_batch_size,
             **engine_build_options,
         )
@@ -151,6 +152,7 @@ def accelerate_with_tensorrt(
             vae_decoder_model,
             create_onnx_path("vae_decoder", onnx_dir, opt=False),
             create_onnx_path("vae_decoder", onnx_dir, opt=True),
+            engine_path=vae_decoder_engine_path,
             opt_batch_size=max_batch_size,
             **engine_build_options,
         )
@@ -162,6 +164,7 @@ def accelerate_with_tensorrt(
             vae_encoder_model,
             create_onnx_path("vae_encoder", onnx_dir, opt=False),
             create_onnx_path("vae_encoder", onnx_dir, opt=True),
+            engine_path=vae_encoder_engine_path,
             opt_batch_size=max_batch_size,
             **engine_build_options,
         )`

engine_path is a bug too, but i got error in compile_vae_decoder (compile_unet fine): RuntimeError: Given groups=1, weight of size [64, 3, 3, 3], expected input[2, 4, 64, 64] to have 3 channels, but got 4 channels instead

@olegchomp
Copy link

got multiple values for keyword argument 'opt_batch_size' means give 'opt_batch_size' more than once, opt_batch_size=max_batch_size and engine_build_options, just del one of them

Sorry but i can you explain solution. I've tried with default example from readme and still get this error #61

did you comments this

opt_batch_size=max_batch_size,

Thank you, now i've got engine_path error as you mentition above

@kulsisme
Copy link

How to resolve it ? I also encountered it :TypeError: streamdiffusion.acceleration.tensorrt.compile_unet() got multiple values for keyword argument 'opt_batch_size'

@rbong
Copy link

rbong commented Dec 28, 2023

I have fixed the batch size and path errors, now I am getting the same error as @8600862

Full error for context:

Exporting model: engines/onnx/vae_decoder.onnx
Traceback (most recent call last):
  File "/home/roger/StreamDiffusion/scripts/run.py", line 30, in <module>
    stream = accelerate_with_tensorrt(
  File "/home/roger/StreamDiffusion/src/streamdiffusion/acceleration/tensorrt/__init__.py", line 153, in accelerate_with_tensorrt
    compile_vae_decoder(
  File "/home/roger/StreamDiffusion/src/streamdiffusion/acceleration/tensorrt/__init__.py", line 56, in compile_vae_decoder
    builder.build(
  File "/home/roger/StreamDiffusion/src/streamdiffusion/acceleration/tensorrt/builder.py", line 54, in build
    export_onnx(
  File "/home/roger/StreamDiffusion/src/streamdiffusion/acceleration/tensorrt/utilities.py", line 418, in export_onnx
    torch.onnx.export(
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 516, in export
    _export(
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 1596, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 1135, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 1011, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 915, in _trace_and_get_graph_from_model
    trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/jit/_trace.py", line 1285, in _get_trace_graph
    outs = ONNXTracedModule(
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/jit/_trace.py", line 133, in forward
    graph, out = torch._C._create_graph_by_tracing(
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/jit/_trace.py", line 124, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1508, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/diffusers-0.24.0-py3.10.egg/diffusers/models/autoencoder_tiny.py", line 331, in forward
    enc = self.encode(sample).latents
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/diffusers-0.24.0-py3.10.egg/diffusers/utils/accelerate_utils.py", line 46, in wrapper
    return method(self, *args, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/diffusers-0.24.0-py3.10.egg/diffusers/models/autoencoder_tiny.py", line 298, in encode
    output = self._tiled_encode(x) if self.use_tiling else self.encoder(x)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1508, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/diffusers-0.24.0-py3.10.egg/diffusers/models/vae.py", line 892, in forward
    x = self.layers(x.add(1).div(2))
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1508, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward
    input = module(input)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1508, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 460, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [64, 3, 3, 3], expected input[2, 4, 64, 64] to have 3 channels, but got 4 channels instead

This appears to be caused by the VAE model class providing the wrong shape of sample input.

    def get_sample_input(self, batch_size, image_height, image_width):
        latent_height, latent_width = self.check_dims(batch_size, image_height, image_width)
        return torch.randn(
            batch_size,

            # FIX
            # 4,
            3,

            latent_height,
            latent_width,
            dtype=torch.float32,
            device=self.device,
        )

I have no idea if the model will actually work with the above changes, because after fixing this issue there appears to be another issue with the model:

Building TensorRT engine for engines/onnx/vae_decoder.opt.onnx: engines/vae_decoder.engine
[E] ModelImporter.cpp:811: While parsing node number 83 [Cast -> "/Cast_output_0"]:
[E] ModelImporter.cpp:812: --- Begin node ---
    input: "/Round_output_0"
    output: "/Cast_output_0"
    name: "/Cast"
    op_type: "Cast"
    attribute {
      name: "to"
      i: 2
      type: INT
    }
[E] ModelImporter.cpp:813: --- End node ---
[E] ModelImporter.cpp:816: ERROR: ModelImporter.cpp:239 In function parseNode:
    [8] Assertion failed: legalUINT8 && "TensorRT does not support UINT8 types for intermediate tensors!"
[E] In node 83 (parseNode): UNSUPPORTED_NODE: Assertion failed: legalUINT8 && "TensorRT does not support UINT8 types for intermediate tensors!"
[!] Could not parse ONNX correctly
Traceback (most recent call last):
  File "/home/roger/StreamDiffusion/scripts/run.py", line 30, in <module>
    stream = accelerate_with_tensorrt(
  File "/home/roger/StreamDiffusion/src/streamdiffusion/acceleration/tensorrt/__init__.py", line 153, in accelerate_with_tensorrt
    compile_vae_decoder(
  File "/home/roger/StreamDiffusion/src/streamdiffusion/acceleration/tensorrt/__init__.py", line 56, in compile_vae_decoder
    builder.build(
  File "/home/roger/StreamDiffusion/src/streamdiffusion/acceleration/tensorrt/builder.py", line 80, in build
    build_engine(
  File "/home/roger/StreamDiffusion/src/streamdiffusion/acceleration/tensorrt/utilities.py", line 393, in build_engine
    engine.build(
  File "/home/roger/StreamDiffusion/src/streamdiffusion/acceleration/tensorrt/utilities.py", line 229, in build
    network_from_onnx_path(onnx_path, flags=[trt.OnnxParserFlag.NATIVE_INSTANCENORM]),
  File "<string>", line 3, in network_from_onnx_path
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/polygraphy/backend/base/loader.py", line 40, in __call__
    return self.call_impl(*args, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/polygraphy/util/util.py", line 694, in wrapped
    return func(*args, **kwargs)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/polygraphy/backend/trt/loader.py", line 211, in call_impl
    trt_util.check_onnx_parser_errors(parser, success)
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/polygraphy/backend/trt/util.py", line 89, in check_onnx_parser_errors
    G_LOGGER.critical("Could not parse ONNX correctly")
  File "/home/roger/StreamDiffusion/.venv/lib/python3.10/site-packages/polygraphy/logger/logger.py", line 597, in critical
    raise PolygraphyException(message) from None
polygraphy.exception.exception.PolygraphyException: Could not parse ONNX correctly

@rbong
Copy link

rbong commented Dec 28, 2023

The tensor shape issue was also reported in #53

They submitted #53 with this fix, it is more likely to work:

    if not os.path.exists(vae_decoder_engine_path):
        # FIX: https://github.com/cumulo-autumn/StreamDiffusion/pull/54/files
        vae.forward = vae.decode

        compile_vae_decoder(
            vae,
            vae_decoder_model,
            create_onnx_path("vae_decoder", onnx_dir, opt=False),
            create_onnx_path("vae_decoder", onnx_dir, opt=True),

            # FIX: SEE: https://github.com/cumulo-autumn/StreamDiffusion/issues/19
            # opt_batch_size=max_batch_size,
            engine_path=vae_decoder_engine_path,

            **engine_build_options,
        )

However I am still getting the "TensorRT does not support UINT8 types for intermediate tensors" error after fixing this.

Worth mentioning I had to tell TensorRT where to find CUDA before running, in case anyone else gets this issue:

export LD_LIBRARY_PATH="`pwd`/.venv/lib/python3.10/site-packages/torch/lib:/usr/lib:/lib"

Note that find "*libcudart*" reveals several libcudart.so files, all of them had this issue.

@rbong
Copy link

rbong commented Dec 28, 2023

TensorRT does not support UINT8 types for intermediate tensors

Turns out to be a bug in CUDA 11.8: NVIDIA/TensorRT#3124

After switching to CUDA 12.1 and making the fixes above (batch size, engine path, vae.forward), TensorRT works.

Make sure you run rm -rf engines/onnx to clear out cached engines.

@hhwang98 hhwang98 mentioned this issue Dec 29, 2023
Open
@hhwang98
Copy link

i fix the bug above (batch size, engine path, vae.forward) ，finally，i get bug：AttributeError: module 'polygraphy.backend.trt.util' has no attribute 'get_bindings_per_profile'
from https://docs.nvidia.com/deeplearning/tensorrt/polygraphy/docs/_modules/polygraphy/backend/trt/util.html, it seem polygraphy.backend.trt.util has no attribute 'get_bindings_per_profile.

@scuizhibin
Copy link

def accelerate_with_tensorrt(
stream: StreamDiffusion,
engine_dir: str,
max_batch_size: int = 2,
min_batch_size: int = 1,
use_cuda_graph: bool = False,
engine_build_options: dict = {},
):
# if "opt_batch_size" not in engine_build_options or engine_build_options["opt_batch_size"] is None:
# engine_build_options["opt_batch_size"] = max_batch_size
text_encoder = stream.text_encoder
unet = stream.unet
vae = stream.vae
#########
if not os.path.exists(unet_engine_path):
compile_unet(
unet,
unet_model,
create_onnx_path("unet", onnx_dir, opt=False),
create_onnx_path("unet", onnx_dir, opt=True),
opt_batch_size=max_batch_size,
engine_path=unet_engine_path,
**engine_build_options,
)

---------------can work--------------------

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@rbong @olegchomp @8600862 @scuizhibin @ngc-shj @bryanhpchiang @hhwang98 @lamquangtuong @kulsisme