# OpenVINO

OpenVINO is an open-source toolkit for optimizing and deploying deep learning models.

- Compiles models for your hardware.
- Supports Linux and Windows.
- Supports CPU, GPU, iGPU, and NPU targets.
- Supports Intel dGPUs, iGPUs, and NPUs.
- Supports AMD dGPUs and iGPUs on Windows with FP16 support.
- Supports NVIDIA GPUs.
- Supports any GPU with OpenCL support.
- Supports CPUs with BF16, FP16, or FP32 support.
- Supports multi-device execution using HETERO mode.

In practice, it is an alternative to TensorRT or Olive that can run on a wide range of hardware.


## Installation

### Preparations

- Install the drivers for your device
- Install [Git and Python](Installation#install-python-and-git)

> [!NOTE]
> Do not mix OpenVINO with your old install. Treat OpenVINO as a separate backend.  

### Running SD.Next with OpenVINO

Open a terminal in the folder where you want to install SD.Next, then clone the repository:

```shell
git clone https://github.com/vladmandic/sdnext
```

Then enter into the sdnext folder:

```shell
cd sdnext
```

Then start WebUI with this command:

Windows:

```shell
.\webui.bat --use-openvino
```

Linux:

```shell
./webui.sh --use-openvino
```

> [!NOTE]
> It will install the necessary libraries at the first run so it will take a while depending on your internet.

## Running SD.Next with Docker

See the [Docker wiki](https://github.com/vladmandic/sdnext/wiki/Docker) if you want to build a custom image.

To run a prebuilt Docker image:

```shell
export SDNEXT_DOCKER_ROOT_FOLDER=~/sdnext
sudo docker run -it \
  --name sdnext-openvino \
  --device /dev/dri \
  -p 7860:7860 \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/app:/app \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/python:/mnt/python \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/data:/mnt/data \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/models:/mnt/models \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/huggingface:/root/.cache/huggingface \
  disty0/sdnext-openvino:latest
```

> [!NOTE]
> It installs required libraries on first run, so startup can take a while depending on your connection.
> Resulting docker image will use 1.1 GB disk space (uncompressed) for the docker image and 2.5 GB for the venv.


## More Info

### Limitations

- Most TensorRT/Olive limitations also apply here.
- Compilation takes a few minutes and using LoRas will trigger recompilation.
- Attention Slicing and HyperTile will not work.  


## Custom Devices

Use the `OpenVINO devices` option in `Backend Settings` to select a device.
Selecting multiple devices combines them into a single `HETERO` device.

If no device is specified, OpenVINO auto-selects the default device.

## Model Caching

OpenVINO can save compiled models to a cache folder so they do not need to be compiled again.

- `OpenVINO disable model caching` in **Backend Settings** turns caching off.
- `Directory for OpenVINO cache` in **System Paths** sets the cache location.