# ONNX Runtime

SD.Next includes support for ONNX Runtime.

## How To Use

`--use-directml` is currently not available because `torch-directml` is not released for the latest PyTorch.
This does not prevent use of `DmlExecutionProvider`.

Set `Diffusers pipeline` to `ONNX Stable Diffusion` on the `System` tab.

## Performance

The performance depends on the execution provider.

## Execution Providers

Currently, `CUDAExecutionProvider` and `DmlExecutionProvider` are supported.

| Provider | ONNX | Olive | GPU | CPU |
| --- | --- | --- | --- | --- |
| CPUExecutionProvider | ✅ | ❌ | ❌ | ✅ |
| DmlExecutionProvider | ✅ | ✅ | ✅ | ❌ |
| CUDAExecutionProvider | ✅ | ✅ | ✅ | ❌ |
| ROCMExecutionProvider | ✅ | 🚧 | ✅ | ❌ |
| OpenVINOExecutionProvider | ✅ | ✅ | ✅ | ✅ |

### CPUExecutionProvider

Not recommended.

Enabled by default.

### DmlExecutionProvider

You can select `DmlExecutionProvider` by installing `onnxruntime-directml`.

DirectX 12 is required (Windows or WSL).

### CUDAExecutionProvider

You can select `CUDAExecutionProvider` by installing `onnxruntime-gpu` (it may already be installed).

### 🚧 ROCMExecutionProvider

Olive for ROCm is working in progress.

### 🚧 OpenVINOExecutionProvider

Under development.

## Supported Features

- Models from huggingface
- Hires and second pass (without sdxl refiner)
- .safetensors VAE

## Known issues

- SD Inpaint may not work.
- SD Upscale pipeline is not tested.
- SDXL Refiner does not work. (due to onnxruntime's issue)


## FAQ

### I'm getting `OnnxStableDiffusionPipeline.__init__() missing 4 required positional arguments: 'vae_encoder', 'vae_decoder', 'text_encoder', and 'unet'`

This is usually caused by a broken model cache generated by a failed conversion or Olive run.
Remove the affected cache in `models/ONNX/cache`.
You can also manage cache from the `ONNX` tab in the UI (enable it in settings if hidden).

## Olive

Olive is a hardware-aware model optimization tool that combines compression, optimization, and compilation (from [PyPI](https://pypi.org/project/olive-ai/)).

### Olive Setup

As Olive optimizes the models in ONNX format, you should set up ONNX Runtime first.

1. Go to `System` tab → `Compute Settings`.
2. Select `Model`, `Text Encoder` and `VAE` in `Compile Model`.
3. Set `Model compile backend` to `olive-ai`.

Olive-specific settings are under `Olive` in `Compute Settings`.

#### How to switch to Olive from torch-directml

Run these commands using PowerShell.

```shell
.\venv\Scripts\activate
pip uninstall torch-directml
pip install torch torchvision --upgrade
pip install onnxruntime-directml
.\webui.bat
```

#### From checkpoint

Model optimization occurs automatically before generation.

Supported model inputs include `.safetensors`, `.ckpt`, and Diffusers pretrained models.
Optimization time depends on your system and execution provider.

The optimized models are automatically cached and used later to create images of the same size (height and width).

#### From Huggingface

If your system does not have enough memory for local optimization, or you want to skip local optimization, download an optimized model from Hugging Face.

Go to `Models` → `Huggingface` tab and download optimized model.

### Advanced Usage

#### Customize Olive workflow

TBA

### Olive Performance

| Property        | Value                                                                                   |
| --------------- | --------------------------------------------------------------------------------------- |
| Prompt          | a castle, best quality                                                                  |
| Negative Prompt | worst quality                                                                           |
| Sampler         | Euler                                                                                   |
| Sampling Steps  | 20                                                                                      |
| Device          | RX 7900 XTX 24GB                                                                        |
| Version         | olive-ai(0.4.0) onnxruntime-directml(1.16.3) ROCm(5.6) torch(olive: 2.1.2, rocm: 2.1.0) |
| Model           | runwayml/stable-diffusion-v1-5 (ROCm), lshqqytiger/stable-diffusion-v1-5-olive (Olive)  |
| Precision       | fp16                                                                                    |
| Token Merging   | Olive(0, not supported) ROCm(0.5)                                                       |

| Olive with DmlExecutionProvider                                                                     | ROCm                                                                                               |
| --------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| ![Olive](https://github.com/vladmandic/sdnext/assets/39524005/4d440319-a0f1-44fe-9772-9c118fdf06ac) | ![ROCm](https://github.com/vladmandic/sdnext/assets/39524005/365eeae9-e666-4641-9333-0aa79dd43ddf) |

### Pros and Cons

#### Pros

- The generation is faster.
- Uses less graphics memory.

#### Cons

- Optimization is required for every models and image sizes.
- Some features are unavailable.

### Olive FAQ

#### My execution provider does not show up in my settings

After activating python venv, run this command and try again:

```shell
(venv) $ pip uninstall onnxruntime onnxruntime-... -y
```