HiRes Fix & Upscaler error: Input type (float) and bias type (struct c10::Half) should be the same

[AugmentedRealityCat]

When using Tiled VAE together with HiRes Fix, the image generation process will stop before completion if you select an Upscaler model (like Ultrasharp, foolhardy_Remacri or ESRGAN) to drive the HiRes Fix.

The same image generation process will work properly if you are select the Latent option (or any of its variants like Latent_bicubic) instead of a proper upscaler.

A similar issue has been flagged in august over here: #292 (comment)

As suggested at the end of that discussion thread, using --no-half-vae can be used to bypass the problem, but this has a negative impact on the overall performance of the A1111-WebUI.

In the log below, you can see the initial launch process followed by two image generation events. The first one uses Latent and was completed successfully. The second one used Ultrasharp as an upscaler and it fails at 50%, exactly when the HiRes Fix process should begin.

venv "C:\stable-diffusion-webui\venvxformers\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.6.0-298-g5e80d9ee
Commit hash: 5e80d9ee99c5899e5e2b130408ffb65a0585a62a
Launching Web UI with arguments: --xformers --api
Civitai Helper: Get Custom Model Folder
Using cache found in C:\Users\User/.cache\torch\hub\isl-org_ZoeDepth_main
img_size [384, 512]
Using cache found in C:\Users\User/.cache\torch\hub\intel-isl_MiDaS_master
Params passed to Resize transform:
        width:  512
        height:  384
        resize_target:  True
        keep_aspect_ratio:  True
        ensure_multiple_of:  32
        resize_method:  minimal
Using pretrained resource url::https://github.com/isl-org/ZoeDepth/releases/download/v1.0/ZoeD_M12_N.pt
Loaded successfully
2023-11-09 01:53:46,069 - ControlNet - INFO - ControlNet v1.1.416
ControlNet preprocessor location: C:\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads
2023-11-09 01:53:46,136 - ControlNet - INFO - ControlNet v1.1.416
[sd-webui-freeu] Controlnet support: *enabled*
[Vec. CC] Style Sheet Loaded...
Loading weights [f848846bcd] from C:\stable-diffusion-webui\models\Stable-diffusion\S_models_selection\modernDisneyXL_v2.safetensors
add tab
*Deforum ControlNet support: enabled*
Creating model from config: C:\stable-diffusion-webui\repositories\generative-models\configs\inference\sd_xl_base.yaml
2023-11-09 01:53:48,281 - modelscope - INFO - PyTorch version 2.0.1+cu118 Found.
2023-11-09 01:53:48,282 - modelscope - INFO - Loading ast index from C:\Users\User\.cache\modelscope\ast_indexer
2023-11-09 01:53:48,346 - modelscope - INFO - Loading done! Current index file version is 1.9.4, with md5 06367ca1744b94bc7be1248cc4c4f509 and a total number of 945 components indexed
[['C:\\stable-diffusion-webui\\extensions\\facechain/resources/inpaint_template\\1.jpg'], ['C:\\stable-diffusion-webui\\extensions\\facechain/resources/inpaint_template\\2.jpg'], ['C:\\stable-diffusion-webui\\extensions\\facechain/resources/inpaint_template\\3.jpg'], ['C:\\stable-diffusion-webui\\extensions\\facechain/resources/inpaint_template\\4.jpg'], ['C:\\stable-diffusion-webui\\extensions\\facechain/resources/inpaint_template\\5.jpg']]
[]
Loading VAE weights specified in settings: C:\stable-diffusion-webui\models\VAE\sdxl_vae.safetensors
Applying attention optimization: sdp-no-mem... done.
Model loaded in 4.8s (load weights from disk: 0.8s, create model: 0.4s, apply weights to model: 2.5s, load VAE: 0.1s, load textual inversion embeddings: 0.8s, calculate empty prompt: 0.1s).
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 18.0s (prepare environment: 1.7s, import torch: 1.6s, import gradio: 0.5s, setup paths: 0.4s, initialize shared: 0.1s, other imports: 0.3s, list SD models: 1.0s, load scripts: 5.6s, create ui: 6.3s, gradio launch: 0.3s).
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:05<00:00,  3.58it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:49<00:00,  2.47s/it]
[Tiled VAE]: input_size: torch.Size([1, 4, 176, 352]), tile_size: 96, padding: 11██████| 40/40 [00:55<00:00, 11.11s/it]
[Tiled VAE]: split to 2x4 = 8 tiles. Optimal tile size 96x96, original tile size 96x96
[Tiled VAE]: Fast mode enabled, estimating group norm parameters on 96 x 48 image
[Tiled VAE]: Executing Decoder Task Queue: 100%|███████████████████████████████████| 984/984 [00:00<00:00, 1133.61it/s]
[Tiled VAE]: Done in 1.493s, max VRAM alloc 10330.902 MB
Total progress: 100%|██████████████████████████████████████████████████████████████████| 40/40 [00:58<00:00,  1.45s/it]
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:03<00:00,  6.22it/s]
[Tiled VAE]: input_size: torch.Size([1, 4, 88, 176]), tile_size: 96, padding: 11       | 20/40 [00:03<00:03,  6.23it/s]
[Tiled VAE]: split to 1x2 = 2 tiles. Optimal tile size 96x96, original tile size 96x96
[Tiled VAE]: Fast mode enabled, estimating group norm parameters on 96 x 48 image
[Tiled VAE]: Executing Decoder Task Queue: 100%|███████████████████████████████████| 246/246 [00:00<00:00, 1230.00it/s]
[Tiled VAE]: Done in 0.723s, max VRAM alloc 9790.335 MB
[Tiled VAE]: input_size: torch.Size([1, 3, 1408, 2816]), tile_size: 1024, padding: 32
[Tiled VAE]: split to 2x3 = 6 tiles. Optimal tile size 928x672, original tile size 1024x1024
[Tiled VAE]: Fast mode enabled, estimating group norm parameters on 1024 x 512 image
[Tiled VAE]: Executing Encoder Task Queue: 100%|███████████████████████████████████| 546/546 [00:00<00:00, 1456.00it/s]
[Tiled VAE]: Done in 1.086s, max VRAM alloc 9512.625 MB
*** Error completing request
*** Arguments: ('task(6f2ajjqhkg7en23)', 'a rock forest', '', [], 20, 'DPM++ 2M Karras', 1, 1, 7, 704, 1408, True, 0.7, 2, '4x-UltraSharp', 0, 0, 0, 'Use same checkpoint', 'Use same sampler', '', '', [], <gradio.routes.Request object at 0x000001F6079FFF40>, 0, False, '', 0.8, 3118465622, False, -1, 0, 0, 0, False, False, False, False, 'base', False, True, False, 0, -1, False, 'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 4, 'None', 2, False, 10, 1, 1, 64, False, False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, True, 1024, 96, True, True, True, False, True, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False, False, 0, 'Gustavosta/MagicPrompt-Stable-Diffusion', '', <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001F38CD62800>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001F607A0DA80>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001F607963E50>, False, 0, 1, 0, 'Version 2', 1.2, 0.9, 0, 0.5, 0, 1, 1.4, 0.2, 0, 0.5, 0, 1, 1, 1, 0, 0.5, 0, 1, True, False, False, False, 0, 0, 1, 0, 0, 0, False, 'Straight Abs.', 'Flat', False, 0.75, 1, False, False, 'positive', 'comma', 0, False, False, 'start', '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False, None, None, False, None, None, False, None, None, False, 50, 7, 1.5, True, '16bpc', '.tiff', 1.2) {}
    Traceback (most recent call last):
      File "C:\stable-diffusion-webui\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "C:\stable-diffusion-webui\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "C:\stable-diffusion-webui\modules\txt2img.py", line 55, in txt2img
        processed = processing.process_images(p)
      File "C:\stable-diffusion-webui\modules\processing.py", line 734, in process_images
        res = process_images_inner(p)
      File "C:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
        return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
      File "C:\stable-diffusion-webui\modules\processing.py", line 869, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "C:\stable-diffusion-webui\modules\processing.py", line 1161, in sample
        return self.sample_hr_pass(samples, decoded_samples, seeds, subseeds, subseed_strength, prompts)
      File "C:\stable-diffusion-webui\modules\processing.py", line 1221, in sample_hr_pass
        samples = images_tensor_to_samples(decoded_samples, approximation_indexes.get(opts.sd_vae_encode_method))
      File "C:\stable-diffusion-webui\modules\sd_samplers_common.py", line 110, in images_tensor_to_samples
        x_latent = model.get_first_stage_encoding(model.encode_first_stage(image))
      File "C:\stable-diffusion-webui\venvxformers\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\generative-models\sgm\models\diffusion.py", line 127, in encode_first_stage
        z = self.first_stage_model.encode(x)
      File "C:\stable-diffusion-webui\repositories\generative-models\sgm\models\autoencoder.py", line 321, in encode
        return super().encode(x).sample()
      File "C:\stable-diffusion-webui\repositories\generative-models\sgm\models\autoencoder.py", line 309, in encode
        moments = self.quant_conv(h)
      File "C:\stable-diffusion-webui\venvxformers\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\extensions-builtin\Lora\networks.py", line 487, in network_Conv2d_forward
        return originals.Conv2d_forward(self, input)
      File "C:\stable-diffusion-webui\venvxformers\lib\site-packages\torch\nn\modules\conv.py", line 463, in forward
        return self._conv_forward(input, self.weight, self.bias)
      File "C:\stable-diffusion-webui\venvxformers\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward
        return F.conv2d(input, weight, bias, self.stride,
    RuntimeError: Input type (float) and bias type (struct c10::Half) should be the same

---

[Quantumdrone]

Same problem. As Augmented said, one has to enable --no-half-vae, which doubles the processing time of generating images in order to not have this error. As a result, it now takes >4 minutes for a single image to gen with a 2x upscale on my 3080.

I've already waited four months to try and use this extension with SDXL. I guess I may never be able to.

[fadedninna]

I'm having the same problem

[clayne]

AUTOMATIC1111/stable-diffusion-webui#12205