thoughts on zluda?

[ride5k]

https://github.com/vosen/ZLUDA

might we finally be able to dump dml??

[lshqqytiger]

Don't look forward to it too much.
I had a look at it. It does run pytorch with cuda.
But because it does not support at least matmul at this moment (on Windows), we can't replace DirectML with it.
Right now, we don't know whether it is faster than DirectML or not.

[lshqqytiger]

I made it work. It is much faster than expectation.
Rough guide:

1. Install AMD HIP SDK
2. Download and unzip zluda from here and place it wherever you want
3. Install torch+cu118
4. Replace venv\Lib\site-packages\torch\lib\cublas64_11.dll with ZLUDA\cublas.dll
   Replace venv\Lib\site-packages\torch\lib\cusparse64_11.dll with ZLUDA\cusparse.dll
5. Add HIP SDK and zluda directory to Path
6. Run webui with cuDNN disabled or use SD.Next

[ride5k]

@lshqqytiger one question i have is how to accomplish step 6: "Run webui with cuDNN disabled." i cannot find any references in the command line arguments? your help is appreciated.

[lshqqytiger]

Add this line somewhere

torch.backends.cudnn.enabled = False

shared_init.py will be appropriate, I think.

[Artiem123]

i tried going to the directories listed but couldn't find the files, i think i'm doing something wrong

[VeteranXT]

stable-diffusion-webui-directml\modules\shared-init.py is where is located. Hope it helps.

[ride5k]

I made it work. It is much faster than expectation. Rough guide:

very interesting, and thank you very much for investigating! i will take a stab at it with my rudimentary skills, and update the thread.

[darthalex2014]

amd rx 580
rocBLAS error: Cannot read C:\Program Files\AMD\ROCm\5.7\bin/rocblas/library/TensileLibrary.dat: No such file or directory for GPU arch : gfx803

rocBLAS error: Could not initialize Tensile host:
regex_error(error_backref): The expression contained an invalid back reference.

guide for newbie. (If I've got it right)
0) Git https://git-scm.com/download/win (64-bit Git for Windows Setup.)
0.1) Python https://www.python.org/ftp/python/3.10.11/python-3.10.11-amd64.exe (During the installation process, make sure to check the box for "Use Git from the Windows Command line and also from 3rd-party-software" to add Git to your system's PATH.)
0.2) SD.Next https://github.com/vladmandic/automatic/archive/refs/heads/dev.zip
0.3) k-diffusion https://github.com/crowsonkb/k-diffusion/archive/refs/heads/master.zip (to automatic-master\modules\k-diffusion)
0.4) zluda https://github.com/lshqqytiger/ZLUDA/releases/download/v3.1-win/ZLUDA.zip
0.5) HIP https://download.amd.com/developer/eula/rocm-hub/AMD-Software-PRO-Edition-23.Q4-Win10-Win11-For-HIP.exe

1. run SD.Next with webui.bat, close and run webui.bat --use-directml, also close.
2. in the automatic-master folder.
   .\venv\Scripts\activate
   pip uninstall torch torchvision torch-directml -y
   pip install torch==2.2.0 torchvision --index-url https://download.pytorch.org/whl/cu118
3. the zluda.zip files into automatic-master/venv/lib/site-packages/torch/lib.
4. Added Path to zluda and AMD HIP SDK
   C:\Program Files\AMD\ROCm\5.7\bin (or your path to ROCm)
   C:\zluda\ (or your path to zluda)
5. run webui.bat

[lshqqytiger]

Using CPU-only Torch is not important. It is just because you don't have nvidia-smi.exe. Find Device=....

[wongdi]

ROCm need redeon pro vii or rx6600+

[Artiem123]

i was trying to follow this but i'm honestly so lost

[Artiem123]

where am i supposed to add these lines ? i was able to find the file but should i just paste it there ?
pip uninstall torch torchvision torch-directml -y
pip install torch==2.2.0 torchvision --index-url https://download.pytorch.org/whl/cu118

[Siralob]

I found on similar discussion on reddit
thanks for navigate me to here btw ;)

How about this one? (gfx803, gfx900, gfx906 ...) I hope this works on every polaris cards.
ROCm-For-All-Windows-5.7.1 means ROCm 5.7.1 and not the ROCm 6.0?

https://github.com/Na3MnO4/ROCmLibs-Fallback

I found some users tested with blender3 later (maybe 4.0) and rendered successfully on windows. so it may works with SD, too.

[darthalex2014]

Go to discord sd next (search in google reddit) download zip file kobald for rx6700. Sorry for english. пт, 16 февр. 2024 г., 05:19 Nikos ***@***.***>:

…

Followed every step from your reply for SD.Next and @lshqqytiger <https://github.com/lshqqytiger> 's reply which resulted in the second error below and i get the same error too with my 6700XT: rocBLAS error: Cannot read C:\Program Files\AMD\ROCm\5.7\bin\/rocblas/library/TensileLibrary.dat: No such file or directory for GPU arch : gfx1031 rocBLAS error: Could not initialize Tensile host: regex_error(error_backref): The expression contained an invalid back reference. Also, only SD.Next seems to work while this webui just stops with this error: venv "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\venv\Scripts\Python.exe" fatal: No names found, cannot describe anything. Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] Version: 1.7.0 Commit hash: 835ee20 no module 'xformers'. Processing without... no module 'xformers'. Processing without... No module 'xformers'. Proceeding without it. D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\venv\lib\site-packages\pytorch_lightning\utilities\distributed.py:258: LightningDeprecationWarning: `pytorch_lightning.utilities.distributed.rank_zero_only` has been deprecated in v1.8.1 and will be removed in v2.0.0. You can import it from `pytorch_lightning.utilities` instead. rank_zero_deprecation( 2024-02-16 02:16:36.015885: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2024-02-16 02:16:36.529858: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. Olive: Failed to load olive-ai: Failed to import transformers.models.gpt2.modeling_tf_gpt2 because of the following error (look up to see its traceback): No module named 'keras.__internal__' Launching Web UI with arguments: Style database not found: D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\styles.csv D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\venv\lib\site-packages\diffusers\utils\outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. torch.utils._pytree._register_pytree_node( Traceback (most recent call last): File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\venv\lib\site-packages\transformers\utils\import_utils.py", line 1086, in _get_module return importlib.import_module("." + module_name, self.__name__) File "C:\Users\Nikos\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1050, in _gcd_import File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 688, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\venv\lib\site-packages\transformers\modeling_tf_utils.py", line 76, in <module> from keras.__internal__ import KerasTensor ModuleNotFoundError: No module named 'keras.__internal__' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\venv\lib\site-packages\transformers\utils\import_utils.py", line 1086, in _get_module return importlib.import_module("." + module_name, self.__name__) File "C:\Users\Nikos\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1050, in _gcd_import File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 688, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\venv\lib\site-packages\optimum\onnxruntime\modeling_diffusion.py", line 41, in <module> from ..exporters.onnx import main_export File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\venv\lib\site-packages\optimum\exporters\__init__.py", line 16, in <module> from .tasks import TasksManager # noqa File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\venv\lib\site-packages\optimum\exporters\tasks.py", line 52, in <module> from transformers import TFPreTrainedModel File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\venv\lib\site-packages\transformers\utils\import_utils.py", line 1076, in __getattr__ module = self._get_module(self._class_to_module[name]) File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\venv\lib\site-packages\transformers\utils\import_utils.py", line 1088, in _get_module raise RuntimeError( RuntimeError: Failed to import transformers.modeling_tf_utils because of the following error (look up to see its traceback): No module named 'keras.__internal__' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\launch.py", line 48, in <module> main() File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\launch.py", line 44, in main start() File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\modules\launch_utils.py", line 677, in start import webui File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\webui.py", line 13, in <module> initialize.imports() File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\modules\initialize.py", line 34, in imports shared_init.initialize() File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\modules\shared_init.py", line 59, in initialize initialize_onnx() File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\modules\onnx_impl\__init__.py", line 236, in initialize from .pipelines.onnx_stable_diffusion_xl_pipeline import OnnxStableDiffusionXLPipeline File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\modules\onnx_impl\pipelines\onnx_stable_diffusion_xl_pipeline.py", line 8, in <module> class OnnxStableDiffusionXLPipeline(CallablePipelineBase, optimum.onnxruntime.ORTStableDiffusionXLPipeline): File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\venv\lib\site-packages\transformers\utils\import_utils.py", line 1076, in __getattr__ module = self._get_module(self._class_to_module[name]) File "D:\AI\Applications\Stable Diffusion (ZLUDA)\stable-diffusion-webui-directml\venv\lib\site-packages\transformers\utils\import_utils.py", line 1088, in _get_module raise RuntimeError( RuntimeError: Failed to import optimum.onnxruntime.modeling_diffusion because of the following error (look up to see its traceback): Failed to import transformers.modeling_tf_utils because of the following error (look up to see its traceback): No module named 'keras.__internal__' Press any key to continue . . . — Reply to this email directly, view it on GitHub <#385 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMHOF6YJK3DTJXJQPJVFL73YT2QYNAVCNFSM6AAAAABDGZ3QKSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DIOBWHA2DG> . You are receiving this because you commented.Message ID: <lshqqytiger/stable-diffusion-webui-directml/repo-discussions/385/comments/8486843 @github.com>

[lshqqytiger]

Possible reasons:

1. AMD HIP SDK is not installed or not in Path.
2. ZLUDA is not in Path.
3. vcruntime140.dll is missing.

[waylaa]

@waylaa Nikos thank you very much for taking the time to share your detailed write-up. I have followed all the steps carefully and successfully and attempted the SDNext repo. However, my initial generation fails with messages similar to @demon8555 and @wxe551 unfortunately.

Hey thanks for all the hard work for getting this to work for us amd users, I'm getting this annoying error but i have the actually files in \venv\lib\site-packages\torch\lib\ directory. Could it be the path setting for Zluda not being correct?

fatal: No names found, cannot describe anything. Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] Version: 1.7.0 Commit hash: 7ed2ff1 Launching Web UI with arguments: --skip-torch-cuda-test --skip-ort Traceback (most recent call last): File "C:\AiServer\stable-diffusion-webui-directml\launch.py", line 48, in main() File "C:\AiServer\stable-diffusion-webui-directml\launch.py", line 44, in main start() File "C:\AiServer\stable-diffusion-webui-directml\modules\launch_utils.py", line 677, in start import webui File "C:\AiServer\stable-diffusion-webui-directml\webui.py", line 13, in initialize.imports() File "C:\AiServer\stable-diffusion-webui-directml\modules\initialize.py", line 14, in imports import torch # noqa: F401 File "C:\AiServer\stable-diffusion-webui-directml\venv\lib\site-packages\torch__init_.py", line 141, in raise err OSError: [WinError 126] The specified module could not be found. Error loading "C:\AiServer\stable-diffusion-webui-directml\venv\lib\site-packages\torch\lib\cublas64_11.dll" or one of its dependencies._

@ride5k @bendybox Yeah, I made a couple mistakes. Follow the guide again.

• I forgot to put the --branch argument when cloning SD.Next. Clone it again.
• I put the link of the main ZLUDA repo instead of lshqqytiger's. Install ZLUDA again.
• Since lshqqytiger's ZLUDA fork is getting updated constantly, it's better to copy and paste the required ZLUDA DLLs to pytorch's lib folder. I updated step 9 in the guide which goes in more detail.
• Try opening webui.bat with .\webui.bat --use-cuda after you finish re-installing it. I don't know if there will be any difference because I don't use --use-cuda and it still works.
• Added several specific options to select in SD.Next's compute settings.

Apparently, there is a ZLUDA guide on SD.Next's wiki.

[ThePixelDiffusionPirate]

Hi @lshqqytiger I read on Discord that you replaced your 5700xt. But if you find a way to use zluda under Win11 with the 5700xt in the near future, can you maybe note it in the wiki or something so that it doesn't get lost? I have no money for a new card^^ big thx 4 your work bro!

[Soulreaver90]

I’m curious on how to replicate those speeds. I have a 6700 XT and have never reached speeds of 43-45it/s. On Linux I max at 6.6xt. With direct ml I’m around 2ish. Just followed your steps and again using the official steps on sd.next, but I max out at around 5ish it/s. All at 512x512

[huangfanglong]

Same issue as bendybox. Setup environment vars already.

This guide helped https://github.com/CS1o/Stable-Diffusion-Info/wiki/Installation-Guides#amd-automatic1111-with-zluda

Stable diffusion launches webui but I am getting error and can't generate

Traceback (most recent call last):
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\gradio\routes.py", line 488, in run_predict
    output = await app.get_blocks().process_api(
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\gradio\blocks.py", line 1431, in process_api
    result = await self.call_function(
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\gradio\blocks.py", line 1103, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\gradio\utils.py", line 707, in wrapper
    response = f(*args, **kwargs)
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\modules\call_queue.py", line 95, in f
    mem_stats = {k: -(v//-(1024*1024)) for k, v in shared.mem_mon.stop().items()}
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\modules\memmon.py", line 99, in stop
    return self.read()
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\modules\memmon.py", line 81, in read
    torch_stats = torch.cuda.memory_stats(self.device)
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\torch\cuda\memory.py", line 258, in memory_stats
    stats = memory_stats_as_nested_dict(device=device)
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\torch\cuda\memory.py", line 270, in memory_stats_as_nested_dict
    return torch._C._cuda_memoryStats(device)
RuntimeError: invalid argument to memory_allocated

[CharlesCato]

Hey guys. I've successfully used zluda (running with a 7900xt on windows).
My only issue for now is:
While generating a 512x768 image with a hiresfix at x1.5 is way faster then with directml but it goes to hell as soon as I try a hiresfix at x2, becoming 14times slower. So basically it goes from 2.19it/s at x1.5 to 7.5s/it at x2.
Anyone as the same issue ? Is it just that zluda is unstable ?
So it's basically faster to gen at hiresfix x2 with directml (for me) while it's 3times faster than directml at 1.5 hiresfix.

[ConsistentFactor]

You have a 7900XT and you're only getting 2.19it/s? I'm getting close to 5it/s on a 7800XT. Something ain't right there (I'm using directML)

[AwesomeApple12]

my 6800 non xt is getting 5.2it/s, its okay speeds, but in ubuntu with ROCm, I can get 8-9it/s.

[ride5k]

unfortunately my GPU is not compatible with zluda, so i cannot contribute to the discussion any further.

[ride5k]

looks like someone has graciously compiled a library for the 6700 https://github.com/brknsoul/ROCmLibs

i will try again!

[ride5k]

was able to get it working, and saw speed increase of about 2.5x!

very pleased so far.

[DivanoDova]

Weird thing for me on RX 6600 generation goes fast (faster than DML) but get stuck at 95% progression for a minute or 2, the only way i found to prevent this is to change VAE Decoder to TAESD but the result look really bad with this option.

[pinea00]

This is super important. RX6600 and RX6700 must use this instead.
Test environment (RX6600 forge) DPM++2M karrs 20step
https://github.com/YellowRoseCx/koboldcpp-rocm/releases/ ---->512512 SD1.5 ＝1.34it/s
Replace it with this
https://github.com/brknsoul/ROCmLibs ---->512512 SD1.5=3it/s

[ride5k]

i was also able to strike medvram precisionfull nohalf flags from args

[pinea00]

Enable FP8 in setting and you don't need any vram args.(forge don't any setting)

[warner1235]

I'm just here to highly recommend Zluda, it simply cut the image generation time in half. AMD RX 5600 XT 6G

[Ming2k8-Coder]

i use 5600xt 6g zluda but it use cpu not gpu, can you share me the output

[VeteranXT]

Output of what? Image?

[CS1o]

Hey, i made Install Guides for AMD on Windows for every common Webui with ZLUDA and DirectML.

Got it working on Auto1111, ComfyUI, Fooocus, with Forge to follow.

To the Install Guides:
https://github.com/CS1o/Stable-Diffusion-Info/wiki/Installation-Guides

[huangfanglong]

Same issue as bendybox. Setup environment vars already.

This guide helped https://github.com/CS1o/Stable-Diffusion-Info/wiki/Installation-Guides#amd-automatic1111-with-zluda

Stable diffusion launches webui but I am getting error and can't generate

Traceback (most recent call last):
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\gradio\routes.py", line 488, in run_predict
    output = await app.get_blocks().process_api(
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\gradio\blocks.py", line 1431, in process_api
    result = await self.call_function(
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\gradio\blocks.py", line 1103, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\gradio\utils.py", line 707, in wrapper
    response = f(*args, **kwargs)
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\modules\call_queue.py", line 95, in f
    mem_stats = {k: -(v//-(1024*1024)) for k, v in shared.mem_mon.stop().items()}
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\modules\memmon.py", line 99, in stop
    return self.read()
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\modules\memmon.py", line 81, in read
    torch_stats = torch.cuda.memory_stats(self.device)
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\torch\cuda\memory.py", line 258, in memory_stats
    stats = memory_stats_as_nested_dict(device=device)
  File "A:\Stable Diffusion\stable-diffusion-webui-amdgpu\venv\lib\site-packages\torch\cuda\memory.py", line 270, in memory_stats_as_nested_dict
    return torch._C._cuda_memoryStats(device)
RuntimeError: invalid argument to memory_allocated

[huangfanglong]

Used the automatic version and guide instead and it works:
https://github.com/vladmandic/automatic/wiki/ZLUDA

And dang, ZLUDA makes such a noticeable improvement in speed. Amazing.

[VeteranXT]

It truly does

[VeteranXT]

@lshqqytiger I really appropriate this zluda and making it work. And i appropriate you for doing all this, I bet many people are but forget to say!