Support for other models?

[cian0]

I tried this one and it didn't work:

https://huggingface.co/SG161222/RealVisXL_V3.0_Turbo/tree/main

How can I load other models? Error is below:

You have disabled the safety checker for <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
The config attributes {'skip_prk_steps': True} were passed to LCMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
Exporting model: engines/thibaud/sdxl_dpo_turbo--lcm_lora-False--tiny_vae-True--max_batch-2--min_batch-2--mode-img2img/unet.engine.onnx
/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py:878: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if dim % default_overall_up_factor != 0:
Traceback (most recent call last):
  File "/home/ian/projs/StreamDiffusion/demo/realtime-img2img/../../utils/wrapper.py", line 546, in _load_model
    compile_unet(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/streamdiffusion/acceleration/tensorrt/__init__.py", line 76, in compile_unet
    builder.build(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/streamdiffusion/acceleration/tensorrt/builder.py", line 54, in build
    export_onnx(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/streamdiffusion/acceleration/tensorrt/utilities.py", line 416, in export_onnx
    torch.onnx.export(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/onnx/utils.py", line 516, in export
    _export(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/onnx/utils.py", line 1596, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/onnx/utils.py", line 1135, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args)
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/onnx/utils.py", line 1011, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/onnx/utils.py", line 915, in _trace_and_get_graph_from_model
    trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/jit/_trace.py", line 1285, in _get_trace_graph
    outs = ONNXTracedModule(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/jit/_trace.py", line 133, in forward
    graph, out = torch._C._create_graph_by_tracing(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/jit/_trace.py", line 124, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1508, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py", line 967, in forward
    if "text_embeds" not in added_cond_kwargs:
TypeError: argument of type 'NoneType' is not iterable
Acceleration has failed. Falling back to normal mode.
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00,  9.37it/s]
You have disabled the safety checker for <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
The config attributes {'skip_prk_steps': True} were passed to LCMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
Exporting model: engines/thibaud/sdxl_dpo_turbo--lcm_lora-False--tiny_vae-True--max_batch-2--min_batch-2--mode-img2img/unet.engine.onnx
Traceback (most recent call last):
  File "/home/ian/projs/StreamDiffusion/demo/realtime-img2img/../../utils/wrapper.py", line 546, in _load_model
    compile_unet(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/streamdiffusion/acceleration/tensorrt/__init__.py", line 76, in compile_unet
    builder.build(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/streamdiffusion/acceleration/tensorrt/builder.py", line 54, in build
    export_onnx(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/streamdiffusion/acceleration/tensorrt/utilities.py", line 416, in export_onnx
    torch.onnx.export(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/onnx/utils.py", line 516, in export
    _export(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/onnx/utils.py", line 1596, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/onnx/utils.py", line 1135, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args)
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/onnx/utils.py", line 1011, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/onnx/utils.py", line 915, in _trace_and_get_graph_from_model
    trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/jit/_trace.py", line 1285, in _get_trace_graph
    outs = ONNXTracedModule(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/jit/_trace.py", line 133, in forward
    graph, out = torch._C._create_graph_by_tracing(
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/jit/_trace.py", line 124, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1508, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/media/ian/extras/condaenvs/streamdiffusion/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py", line 967, in forward
    if "text_embeds" not in added_cond_kwargs:
TypeError: argument of type 'NoneType' is not iterable
Acceleration has failed. Falling back to normal mode.

[leehua03f3]

My guess would be you are trying to run a variation of SDXL Turbo while they are currently only support SD1.5 Turbo (see here). I tried with different variation of SD1.5 and it worked.

[cian0]

Thanks! I figured this is 2.1 sd, any idea how i can merge existing models into this one?