AMD's "18X" Performance with Olive Article

[pw405]

AMD published an article discussing how to generate Olive optimized models that runs 18X faster... anybody try to test this yet?

https://community.amd.com/t5/gaming/how-to-running-optimized-automatic1111-stable-diffusion-webui-on/ba-p/625585

[lshqqytiger]

The article says 9.9x faster, but it does not apply to every AMD gpus.
I can get 1.35x faster speed with RX 5700 XT and lower vram usage.
No reports have been made on the RX 7900 XT(X).
You can try yourself: #149
(the article you attached does not use the Olive optimization function built into this repository. I recommend that you use the built-in Olive optimization feature)

[pw405]

OK, unfortunately I haven't been able to get the Olive/Onyxx running after following the info in #149.

I have a completely separate copy of the entire repo on a seperate disk. Deleted the venv, installed everything fresh. After resolving multiple python library package errors, when when launching WebUI-user.bat, I get the following:

Traceback (most recent call last):
File "G:\STABLE\stable-diffusion-webui-directml\launch.py", line 39, in main()
File "G:\STABLE\stable-diffusion-webui-directml\launch.py", line 35, in main start()
File "G:\STABLE\stable-diffusion-webui-directml\modules\launch_utils.py", line 447, in start
webui.webui()
File "G:\STABLE\stable-diffusion-webui-directml\webui.py", line 383, in webui
initialize()
File "G:\STABLE\stable-diffusion-webui-directml\webui.py", line 264, in initialize
initialize_rest(reload_script_modules=False)
File "G:\STABLE\stable-diffusion-webui-directml\webui.py", line 282, in initialize_rest
modules.sd_models.list_models()
File "G:\STABLE\stable-diffusion-webui-directml\modules\sd_models.py", line 130, in list_models
model_list = [f for f in listdir(onnx_model_path) if os.path.isdir(os.path.join(onnx_model_path, f))]
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'G:\STABLE\stable-diffusion-webui-directml\models\ONNX-Olive'

[lshqqytiger]

Create ONNX-Olive folder under models directory and try again.

[pw405]

I'm an idiot! Helps if I don't spell "ONNX" as "ONXX" !!!

However... I think the root of this issue is some Python library version conflict (and again, I'm certainly not a pro at any of this.)

I started over with a brand new blank/never used repo and after creating the ONNX-Olive folder in Models, when I try to run the upgrades in #149, I think there is a version conflict in requirements_versions.txt and perhaps a .DLL that is failing to register.

After running the commands highlighted in #149:

pip install -r requirements.txt --upgrade
pip install -r requirements_onnx.txt --upgrade
pip install -r requirements_olive.txt --upgrade
pip install torch==1.13.1 torchvision==0.14.1 torch-directml==0.1.13.1.dev230413

The Majority of requirements are satisfied with the exception of:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
optimum 1.11.2 requires transformers[sentencepiece]>=4.26.0, but you have transformers 4.25.1 which is incompatible.
olive-ai 0.2.1 requires numpy<=1.23.4, but you have numpy 1.24.4 which is incompatible.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torch-directml 0.2.0.dev230426 requires torch==2.0.0, but you have torch 1.13.1 which is incompatible.
torch-directml 0.2.0.dev230426 requires torchvision==0.15.1, but you have torchvision 0.14.1 which is incompatible.
olive-ai 0.2.1 requires numpy<=1.23.4, but you have numpy 1.24.4 which is incompatible.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torch-directml 0.2.0.dev230426 requires torch==2.0.0, but you have torch 1.13.1 which is incompatible.
torch-directml 0.2.0.dev230426 requires torchvision==0.15.1, but you have torchvision 0.14.1 which is incompatible.
olive-ai 0.2.1 requires numpy<=1.23.4, but you have numpy 1.24.4 which is incompatible.

However, I'm not sure if it matters, but I noticed the requirements_versions.txt file lists numpy==1.23.5 and torch set to no version.

So I manually run:

Pip install numpy==1.23.4

pip install torch==2.0.0

When Installing torch==2.0.0 I get:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.14.1 requires torch==1.13.1, but you have torch 2.0.0 which is incompatible.
torch-directml 0.1.13.1.dev230413 requires torch==1.13.1, but you have torch 2.0.0 which is incompatible.

Then manually ran the command alone this time - skipping requirements.txt --upgrade, requirements_onnx.txt --upgrade, requirements_olive.txt --upgrade:

pip install torch==1.13.1 torchvision==0.14.1 torch-directml==0.1.13.1.dev230413

No error messages!

Upon launching webui-user.bat with --olive --backend directml --skip-install

I get:

 File "G:\OLIVESD\stable-diffusion-webui-directml\launch.py", line 39, in <module>
    main()
  File "G:\OLIVESD\stable-diffusion-webui-directml\launch.py", line 35, in main
    start()
  File "G:\OLIVESD\stable-diffusion-webui-directml\modules\launch_utils.py", line 443, in start
    import webui
  File "G:\OLIVESD\stable-diffusion-webui-directml\webui.py", line 54, in <module>
    from modules.call_queue import wrap_gradio_gpu_call, wrap_queued_call, queue_lock  # noqa: F401
  File "G:\OLIVESD\stable-diffusion-webui-directml\modules\call_queue.py", line 6, in <module>
    from modules import shared, progress, errors
  File "G:\OLIVESD\stable-diffusion-webui-directml\modules\shared.py", line 22, in <module>
    from ldm.models.diffusion.ddpm import LatentDiffusion
  File "G:\OLIVESD\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 20, in <module>
    from pytorch_lightning.utilities.distributed import rank_zero_only
ModuleNotFoundError: No module named 'pytorch_lightning.utilities.distributed'

Sorry for long post, but I figured if it helps somebody else in the future, it may be worth noting.

[lshqqytiger]

Ignore every version conflict error/warning message. Don't manually upgrade torch and related packages. (olive won't work with torch 2.0.0 or newer)
Run these commands and try again. (ignore all red texts from pip)

$ pip uninstall torch torchvision torch-directml pytorch_lightning numpy
$ pip install -r requirements.txt --upgrade
$ pip install -r requirements_onnx.txt --upgrade
$ pip install -r requirements_olive.txt --upgrade
$ pip install torch==1.13.1 torchvision==0.14.1 torch-directml==0.1.13.1.dev230413

[hisham-hchowdhu]

to get upside on AMD GPU, are using the latest driver from:
AMD Software: Adrenalin Edition™ 23.7.2 or newer (https://www.amd.com/en/support)

[pw405]

After running the above, when I launch webui-user.bat with --olive --backend directml --skip-install It gives the error about Pytorch Lightning again:

   from ldm.models.diffusion.ddpm import LatentDiffusion
  File "G:\OLIVESD\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 20, in <module>
    from pytorch_lightning.utilities.distributed import rank_zero_only
ModuleNotFoundError: No module named 'pytorch_lightning.utilities.distributed'

If I remove --skip-install two errors appear:

Installing Olive
Traceback (most recent call last):
  File "G:\OLIVESD\stable-diffusion-webui-directml\launch.py", line 39, in <module>
    main()
  File "G:\OLIVESD\stable-diffusion-webui-directml\launch.py", line 30, in main
    prepare_environment()
  File "G:\OLIVESD\stable-diffusion-webui-directml\modules\launch_utils.py", line 412, in prepare_environment
    run_pip("install olive-ai[directml]", "Olive")
  File "G:\OLIVESD\stable-diffusion-webui-directml\modules\launch_utils.py", line 137, in run_pip
    return run(f'"{python}" -m pip {command} --prefer-binary{index_url_line}', desc=f"Installing {desc}", errdesc=f"Couldn't install {desc}", live=live)
  File "G:\OLIVESD\stable-diffusion-webui-directml\modules\launch_utils.py", line 114, in run
    raise RuntimeError("\n".join(error_bits))
RuntimeError: Couldn't install Olive.

and the DLL registration failure:

stderr: ERROR: Could not install packages due to an OSError: [WinError 5] Access is denied: 'G:\\OLIVESD\\stable-diffusion-webui-directml\\venv\\Lib\\site-packages\\~0mpy\\.libs\\libopenblas.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.gfortran-win_amd64.dll'
Check the permissions.

When trying to manually register the DLL with RegSvr32, it shows the following error:

[lshqqytiger]

That error is quite strange. Could you let me know the optimization options you used? Probably, there's an invalid path or something.

[pw405]

Optimizations... As in scaled dot product, quad, etc? I haven't set any in this repo, in fact this repo has never even generated an image. Could some of this be related to have Python installed on C: drive and this repo on secondary (G:) drive

[lshqqytiger]

No. I mean, Olive optimization configs. like model path, output path, height, width, etc.

[pw405]

Honestly, didn't pay close enough attention, I tried to put the names in, but the UI wouldn't allow me to select models at all, very odd. I'll try when I get home from work today.

[wxe551]

Try add --ui-debug-mode to arguements, then it won't look for model. works for me.

[songib]

How to setup all of this?
idk much about this stuff. I'm on Win11 with 5700xt.

Update: I got this instead.
Launching Web UI with arguments: --skip-install --olive --backend directml --medvram --no-half --disable-safe-unpickle --skip-torch-cuda-test --always-batch-cond-uncond --disable-nan-check --api --autolaunch --theme dark --use-cpu CLIP BLIP interrogate no module 'xformers'. Processing without... No SDP backend available, likely because you are running in pytorch versions < 2.0. In fact, you are using PyTorch 1.13.1+cpu. You might want to consider upgrading. no module 'xformers'. Processing without... No module 'xformers'. Proceeding without it. ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ *******************************************************\stable-diffusion-webui-directml │ │ Olive\launch.py:39 in <module> │ │ │ │ 36 │ │ 37 │ │ 38 if __name__ == "__main__": │ │ ❱ 39 │ main() │ │ 40 │ │ │ │ *******************************************************\stable-diffusion-webui-directml │ │ Olive\launch.py:35 in main │ │ │ │ 32 │ if args.test_server: │ │ 33 │ │ configure_for_tests() │ │ 34 │ │ │ ❱ 35 │ start() │ │ 36 │ │ 37 │ │ 38 if __name__ == "__main__": │ │ │ │ *******************************************************\stable-diffusion-webui-directml │ │ Olive\modules\launch_utils.py:443 in start │ │ │ │ 440 │ │ 441 def start(): │ │ 442 │ print(f"Launching {'API server' if '--nowebui' in sys.argv else 'Web UI'} with argum │ │ ❱ 443 │ import webui │ │ 444 │ if '--nowebui' in sys.argv: │ │ 445 │ │ webui.api_only() │ │ 446 │ else: │ │ │ │ *******************************************************\stable-diffusion-webui-directml │ │ Olive\webui.py:54 in <module> │ │ │ │ 51 startup_timer.record("import ldm") │ │ 52 │ │ 53 from modules import extra_networks │ │ ❱ 54 from modules.call_queue import wrap_gradio_gpu_call, wrap_queued_call, queue_lock # noq │ │ 55 │ │ 56 # Truncate version number of nightly/local build of PyTorch to not cause exceptions with │ │ 57 if ".dev" in torch.__version__ or "+git" in torch.__version__: │ │ │ │ *******************************************************\stable-diffusion-webui-directml │ │ Olive\modules\call_queue.py:6 in <module> │ │ │ │ 3 import threading │ │ 4 import time │ │ 5 │ │ ❱ 6 from modules import shared, progress, errors │ │ 7 │ │ 8 queue_lock = threading.Lock() │ │ 9 │ │ │ │ *******************************************************\stable-diffusion-webui-directml │ │ Olive\modules\shared.py:72 in <module> │ │ │ │ 69 if cmd_opts.olive: │ │ 70 │ cmd_opts.onnx = True │ │ 71 if cmd_opts.backend == "directml": │ │ ❱ 72 │ directml_init() │ │ 73 │ │ 74 │ │ 75 devices.device, devices.device_interrogate, devices.device_gfpgan, devices.device_esrgan │ │ │ │ *******************************************************\stable-diffusion-webui-directml │ │ Olive\modules\dml\__init__.py:38 in directml_init │ │ │ │ 35 │ torch.cuda.mem_get_info = torch.dml.mem_get_info │ │ 36 │ │ 37 def directml_init(): │ │ ❱ 38 │ from modules.dml.backend import DirectML # pylint: disable=ungrouped-imports │ │ 39 │ # Alternative of torch.cuda for DirectML. │ │ 40 │ torch.dml = DirectML │ │ 41 │ │ │ │ *******************************************************\stable-diffusion-webui-directml │ │ Olive\modules\dml\backend.py:4 in <module> │ │ │ │ 1 # pylint: disable=no-member,no-self-argument,no-method-argument │ │ 2 from typing import Optional, Callable │ │ 3 import torch │ │ ❱ 4 import torch_directml # pylint: disable=import-error │ │ 5 import modules.dml.amp as amp │ │ 6 │ │ 7 from .utils import rDevice, get_device │ │ │ │ *******************************************************\stable-diffusion-webui-directml │ │ Olive\venv\lib\site-packages\torch_directml\__init__.py:16 in <module> │ │ │ │ 13 torch.ops.load_library(directml_dll) │ │ 14 │ │ 15 # import native apis │ │ ❱ 16 import torch_directml_native │ │ 17 │ │ 18 from .device import * │ │ 19 │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ ImportError: DLL load failed while importing torch_directml_native: The specified procedure could not be found. Press any key to continue . . .

[songib]

@lshqqytiger still getting this issue using the local model for optimization
`Unloaded weights 0.0s.

Optimizing safety_checker
*** Error completing request
*** Arguments: photon_v1.safetensors', 'vae-ft-mse-840000-ema-pruned.safetensors', 'vae', 'Photon', 'Photon-512', True, True, False, True, True, True, 'euler', True, 512, False, '', '', '') {}
Traceback (most recent call last):
File "\stable-diffusion-webui-directml Olive\modules\call_queue.py", line 58, in f
res = list(func(*args, **kwargs))
File "\stable-diffusion-webui-directml Olive\modules\call_queue.py", line 37, in f
res = func(*args, **kwargs)
File "\stable-diffusion-webui-directml Olive\modules\sd_olive_ui.py", line 34, in optimize_from_ckpt
optimize(unoptimized_dir, optimized_dir, pipeline, vae_id, vae_subfolder, safety_checker, text_encoder, text_encoder_2, unet, vae_decoder, vae_encoder, use_fp16, sample_size if hw_synced else sample_size[0], sample_size if hw_synced else sample_size[1], olive_merge_lora, *olive_merge_lora_inputs)
File "\stable-diffusion-webui-directml Olive\modules\sd_olive_ui.py", line 98, in optimize
olive_run(olive_config)
File "\stable-diffusion-webui-directml Olive\venv\lib\site-packages\olive\workflows\run\run.py", line 147, in run
engine = config.engine.create_engine()
File "\stable-diffusion-webui-directml Olive\venv\lib\site-packages\olive\workflows\run\config.py", line 51, in create_engine
return Engine(config)
File "\stable-diffusion-webui-directml Olive\venv\lib\site-packages\olive\engine\engine.py", line 131, in init
assert self.accelerator_specs, (
AssertionError: No valid accelerator specified for target system. Please specify the accelerators in the target system or provide valid execution providers. Given execution providers: ['DmlExecutionProvider']. Current accelerators: ['gpu'].Supported execution providers: {'cpu': ['CPUExecutionProvider', 'OpenVINOExecutionProvider'], 'gpu': ['DmlExecutionProvider', 'CUDAExecutionProvider', 'OpenVINOExecutionProvider', 'TensorrtExecutionProvider', 'CPUExecutionProvider'], 'npu': ['QNNExecutionProvider', 'CPUExecutionProvider']}.`

[lshqqytiger]

Run this script and let me know its output.

import onnxruntime as ort
print(ort.get_available_providers())

[songib]

@lshqqytiger ['DmlExecutionProvider', 'CPUExecutionProvider']
Another thing is that I used a symbolic link for the model folder. it may cause some conflict. idk

[lshqqytiger]

Currently, loading vae from .safetensors and .pt file is not supported.
Try again with default. (empty field)

[songib]

@lshqqytiger ahh, ok. ty

[hisham-hchowdhu]

The article says 9.9x faster, but it does not apply to every AMD gpus. I can get 1.35x faster speed with RX 5700 XT and lower vram usage. No reports have been made on the RX 7900 XT(X). You can try yourself: #149 (the article you attached does not use the Olive optimization function built into this repository. I recommend that you use the built-in Olive optimization feature)

AMD published an article discussing how to generate Olive optimized models that runs 18X faster... anybody try to test this yet?

https://community.amd.com/t5/gaming/how-to-running-optimized-automatic1111-stable-diffusion-webui-on/ba-p/625585

to get upside on AMD GPU, are using the latest driver from:
AMD Software: Adrenalin Edition™ 23.7.2 or newer (https://www.amd.com/en/support)

[pw405]

5800X and 7900XTX (Windows 10).

[hisham-hchowdhu]

Please move to Windows11 and try.

https://community.amd.com/t5/gaming/how-to-running-optimized-automatic1111-stable-diffusion-webui-on/ba-p/625585

make sure after section 3 you test the optimized model using Olive before moving to section4(Automatic).
"To test the optimized model, run the following command:
python stable_diffusion.py --interactive --num_images 2"
section3 should show big improvement before you can move to section4(Automatic1111).
both optimized and unoptimized model after section3 should be stored at:
\olive\examples\directml\stable_diffusion\models. make sure optimized models are smaller

[pw405]

Is Windows 11 required? I've been trying on Windows 10 the whole time, however I haven't tried the Anaconda methods outlined in the AMD article. I've been trying to use the UI in the Olive tab of Lshqqytiger's Automatic1111 WebUI.

[lshqqytiger]

When NVIDIA first introduced Olive (although Microsoft developed), they specifically said Windows 11.
I don't know it is required, but it does make sense because there's WDDM which depends on Windows build and includes DirectX.

[hisham-hchowdhu]

AMD collaborates with Microsoft closely on optimizing Olive path including SD use cases. what Olive generates is accelerated in the AMD driver ML layers at runtime. So, without Windows11, some of those accelerations may be missing. Try SD1.5 from Olive branch first to make sure you are getting the perf + expected, then move to using Automatic with Olive

[pw405]

Can confirm!! Successfully getting ~18 iterations/second with 1.5 based models once converted to Olive at 512 X 512 see comment in the About Olive Support (#149) discussion:

#149 (reply in thread)

See important comment from lshqqytiger above mine. I was configuring the Optimize Checkpoint tab wrong, AND I thought I needed to double-convert a model both Optimize Checkpoint and Optimize Onnx Model tab

#149 (reply in thread)