notebooks/dynamicrafter-animating-images/dynamicrafter-animating-images.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a30812de-c46e-44a3-8194-b7f6f0fd4707",
   "metadata": {},
   "source": [
    "# Animating Open-domain Images with DynamiCrafter and OpenVINO\n",
    "\n",
    "Animating a still image offers an engaging visual experience. Traditional image animation techniques mainly focus on animating natural scenes with stochastic dynamics (e.g. clouds and fluid) or domain-specific motions (e.g. human hair or body motions), and thus limits their applicability to more general visual content. To overcome this limitation, [DynamiCrafter team](https://doubiiu.github.io/projects/DynamiCrafter/) explores the synthesis of dynamic content for open-domain images, converting them into animated videos. The key idea is to utilize the motion prior of text-to-video diffusion models by incorporating the image into the generative process as guidance. Given an image, DynamiCrafter team first projects it into a text-aligned rich context representation space using a query transformer, which facilitates the video model to digest the image content in a compatible fashion. However, some visual details still struggle to be preserved in the resultant videos. To supplement with more precise image information, DynamiCrafter team further feeds the full image to the diffusion model by concatenating it with the initial noises. Experimental results show that the proposed method can produce visually convincing and more logical & natural motions, as well as higher conformity to the input image.\n",
    "\n",
    "In this tutorial, we consider how to use DynamiCrafter with OpenVINO. An additional part demonstrates how to run optimization with [NNCF](https://github.com/openvinotoolkit/nncf/) to speed up model.\n",
    "\n",
    "<table class=\"center\">\n",
    "  <tr>\n",
    "    <td colspan=\"2\">\"bear playing guitar happily, snowing\"</td>\n",
    "    <td colspan=\"2\">\"boy walking on the street\"</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "  <td>\n",
    "    <img src=https://github.com/Doubiiu/DynamiCrafter/blob/main/assets/showcase/guitar0.jpeg_00.png?raw=True width=\"170\">\n",
    "  </td>\n",
    "  <td>\n",
    "    <img src=https://github.com/Doubiiu/DynamiCrafter/blob/main/assets/showcase/guitar0.gif?raw=True width=\"170\">\n",
    "  </td>\n",
    "  <td>\n",
    "    <img src=https://github.com/Doubiiu/DynamiCrafter/blob/main/assets/showcase/walk0.png_00.png?raw=True width=\"170\">\n",
    "  </td>\n",
    "  <td>\n",
    "    <img src=https://github.com/Doubiiu/DynamiCrafter/blob/main/assets/showcase/walk0.gif?raw=True width=\"170\">\n",
    "  </td>\n",
    "  </tr>\n",
    "</table >\n",
    "\n",
    "\n",
    "#### Table of contents:\n",
    "- [Prerequisites](#Prerequisites)\n",
    "- [Load the original model](#Load-the-original-model)\n",
    "- [Convert the model to OpenVINO IR](#Convert-the-model-to-OpenVINO-IR)\n",
    "  - [Convert CLIP text encoder](#Convert-CLIP-text-encoder)\n",
    "  - [Convert CLIP image encoder](#Convert-CLIP-image-encoder)\n",
    "  - [Convert AE encoder](#Convert-AE-encoder)\n",
    "  - [Convert Diffusion U-Net model](#Convert-Diffusion-U-Net-model)\n",
    "  - [Convert AE decoder](#Convert-AE-decoder)\n",
    "- [Compiling models](#Compiling-models)\n",
    "- [Building the pipeline](#Building-the-pipeline)\n",
    "- [Run OpenVINO pipeline inference](#Run-OpenVINO-pipeline-inference)\n",
    "- [Quantization](#Quantization)\n",
    "    - [Prepare calibration dataset](#Prepare-calibration-dataset)\n",
    "    - [Run Quantization](#Run-Quantization)\n",
    "    - [Run Weights Compression](#Run-Weights-Compression)\n",
    "    - [Compare model file sizes](#Compare-model-file-sizes)\n",
    "    - [Compare inference time of the FP32 and INT8 pipelines](#Compare-inference-time-of-the-FP32-and-INT8-pipelines)\n",
    "- [Interactive inference](#Interactive-inference)\n",
    "### Installation Instructions\n",
    "\n",
    "This is a self-contained example that relies solely on its own code.\n",
    "\n",
    "We recommend  running the notebook in a virtual environment. You only need a Jupyter server to start.\n",
    "For details, please refer to [Installation Guide](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide).\n",
    "\n",
    "<img referrerpolicy=\"no-referrer-when-downgrade\" src=\"https://static.scarf.sh/a.png?x-pxid=5b5a4db0-7875-4bfb-bdbd-01698b5b1a77&file=notebooks/dynamicrafter-animating-images/dynamicrafter-animating-images.ipynb\" />\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f9dc9580-da81-47dd-b5d3-3cafa8f5a4b5",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eac97b7e-2db7-41b3-8dc4-488c5b5cd275",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q \"openvino>=2024.2.0\" \"nncf>=2.11.0\" \"datasets>=2.20.0\"\n",
    "%pip install -q \"gradio>=4.19\" omegaconf einops pytorch_lightning kornia \"open_clip_torch==2.22.0\" transformers av opencv-python \"torch==2.2.2\" \"av==12.0.0\" --extra-index-url https://download.pytorch.org/whl/cpu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4c8de050-19e7-42a2-bf5a-98ca5eef050e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import requests\n",
    "\n",
    "\n",
    "if not Path(\"cmd_helper.py\").exists():\n",
    "    r = requests.get(\n",
    "        url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/cmd_helper.py\",\n",
    "    )\n",
    "    open(\"cmd_helper.py\", \"w\").write(r.text)\n",
    "\n",
    "if not Path(\"notebook_utils.py\").exists():\n",
    "    r = requests.get(\n",
    "        url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py\",\n",
    "    )\n",
    "    open(\"notebook_utils.py\", \"w\").write(r.text)\n",
    "\n",
    "# Read more about telemetry collection at https://github.com/openvinotoolkit/openvino_notebooks?tab=readme-ov-file#-telemetry\n",
    "from notebook_utils import collect_telemetry\n",
    "\n",
    "collect_telemetry(\"dynamicrafter-animating-images.ipynb\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d5526c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from cmd_helper import clone_repo\n",
    "\n",
    "\n",
    "clone_repo(\"https://github.com/Doubiiu/DynamiCrafter.git\", \"26e665cd6c174234238d2ded661e2e56f875d360\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a3c0c659-aad3-4962-8db7-7b123379f01a",
   "metadata": {},
   "source": [
    "## Load and run the original pipeline\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "We will use model for 256x256 resolution as example. Also, models for 320x512 and 576x1024 are [available](https://github.com/Doubiiu/DynamiCrafter?tab=readme-ov-file#-models)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3ce0481-d7de-4d37-9414-c72dc6488f8e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AE working on z of shape (1, 4, 32, 32) = 4096 dimensions.\n",
      ">>> model checkpoint loaded.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from collections import OrderedDict\n",
    "\n",
    "import torch\n",
    "from huggingface_hub import hf_hub_download\n",
    "from omegaconf import OmegaConf\n",
    "\n",
    "from utils.utils import instantiate_from_config\n",
    "\n",
    "\n",
    "def load_model_checkpoint(model, ckpt):\n",
    "    def load_checkpoint(model, ckpt, full_strict):\n",
    "        state_dict = torch.load(ckpt, map_location=\"cpu\")\n",
    "        if \"state_dict\" in list(state_dict.keys()):\n",
    "            state_dict = state_dict[\"state_dict\"]\n",
    "            try:\n",
    "                model.load_state_dict(state_dict, strict=full_strict)\n",
    "            except Exception:\n",
    "                ## rename the keys for 256x256 model\n",
    "                new_pl_sd = OrderedDict()\n",
    "                for k, v in state_dict.items():\n",
    "                    new_pl_sd[k] = v\n",
    "\n",
    "                for k in list(new_pl_sd.keys()):\n",
    "                    if \"framestride_embed\" in k:\n",
    "                        new_key = k.replace(\"framestride_embed\", \"fps_embedding\")\n",
    "                        new_pl_sd[new_key] = new_pl_sd[k]\n",
    "                        del new_pl_sd[k]\n",
    "                model.load_state_dict(new_pl_sd, strict=full_strict)\n",
    "        else:\n",
    "            ## deepspeed\n",
    "            new_pl_sd = OrderedDict()\n",
    "            for key in state_dict[\"module\"].keys():\n",
    "                new_pl_sd[key[16:]] = state_dict[\"module\"][key]\n",
    "            model.load_state_dict(new_pl_sd, strict=full_strict)\n",
    "\n",
    "        return model\n",
    "\n",
    "    load_checkpoint(model, ckpt, full_strict=True)\n",
    "    print(\">>> model checkpoint loaded.\")\n",
    "    return model\n",
    "\n",
    "\n",
    "def download_model():\n",
    "    REPO_ID = \"Doubiiu/DynamiCrafter\"\n",
    "    if not os.path.exists(\"./checkpoints/dynamicrafter_256_v1/\"):\n",
    "        os.makedirs(\"./checkpoints/dynamicrafter_256_v1/\")\n",
    "    local_file = os.path.join(\"./checkpoints/dynamicrafter_256_v1/model.ckpt\")\n",
    "    if not os.path.exists(local_file):\n",
    "        hf_hub_download(repo_id=REPO_ID, filename=\"model.ckpt\", local_dir=\"./checkpoints/dynamicrafter_256_v1/\", local_dir_use_symlinks=False)\n",
    "\n",
    "    ckpt_path = \"checkpoints/dynamicrafter_256_v1/model.ckpt\"\n",
    "    config_file = \"DynamiCrafter/configs/inference_256_v1.0.yaml\"\n",
    "    config = OmegaConf.load(config_file)\n",
    "    model_config = config.pop(\"model\", OmegaConf.create())\n",
    "    model_config[\"params\"][\"unet_config\"][\"params\"][\"use_checkpoint\"] = False\n",
    "    model = instantiate_from_config(model_config)\n",
    "    model = load_model_checkpoint(model, ckpt_path)\n",
    "    model.eval()\n",
    "\n",
    "    return model\n",
    "\n",
    "\n",
    "model = download_model()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "be9643c8-a70c-4dba-8259-d4467ae82949",
   "metadata": {},
   "source": [
    "## Convert the model to OpenVINO IR\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Let's define the conversion function for PyTorch modules. We use `ov.convert_model` function to obtain OpenVINO Intermediate Representation object and `ov.save_model` function to save it as XML file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1bf5ffa9-c7d5-4485-915c-48ed18f657dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import gc\n",
    "\n",
    "import openvino as ov\n",
    "\n",
    "\n",
    "def convert(model: torch.nn.Module, xml_path: str, example_input, input_shape=None):\n",
    "    xml_path = Path(xml_path)\n",
    "    if not xml_path.exists():\n",
    "        xml_path.parent.mkdir(parents=True, exist_ok=True)\n",
    "        with torch.no_grad():\n",
    "            if not input_shape:\n",
    "                converted_model = ov.convert_model(model, example_input=example_input)\n",
    "            else:\n",
    "                converted_model = ov.convert_model(model, example_input=example_input, input=input_shape)\n",
    "        ov.save_model(converted_model, xml_path, compress_to_fp16=False)\n",
    "\n",
    "        # cleanup memory\n",
    "        torch._C._jit_clear_class_registry()\n",
    "        torch.jit._recursive.concrete_type_store = torch.jit._recursive.ConcreteTypeStore()\n",
    "        torch.jit._state._clear_class_state()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3c63518d-957d-4358-8711-cf6fb935d8be",
   "metadata": {},
   "source": [
    "Flowchart of DynamiCrafter proposed in [the paper](https://arxiv.org/abs/2310.12190):\n",
    "\n",
    "![schema](https://github.com/openvinotoolkit/openvino_notebooks/assets/76171391/d1033876-c664-4345-a254-0649edbf1906)\n",
    "Description:\n",
    "> During inference, our model can generate animation clips from noise conditioned on the input still image.\n",
    "\n",
    "Let's convert models from the pipeline one by one."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d9824415cd5b0ffd",
   "metadata": {},
   "source": [
    "### Convert CLIP text encoder\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e0b1637-7757-49bb-b60f-175f7d77b470",
   "metadata": {},
   "outputs": [],
   "source": [
    "from lvdm.modules.encoders.condition import FrozenOpenCLIPEmbedder\n",
    "\n",
    "MODEL_DIR = Path(\"models\")\n",
    "\n",
    "COND_STAGE_MODEL_OV_PATH = MODEL_DIR / \"cond_stage_model.xml\"\n",
    "\n",
    "\n",
    "class FrozenOpenCLIPEmbedderWrapper(FrozenOpenCLIPEmbedder):\n",
    "    def forward(self, tokens):\n",
    "        z = self.encode_with_transformer(tokens.to(self.device))\n",
    "        return z\n",
    "\n",
    "\n",
    "cond_stage_model = FrozenOpenCLIPEmbedderWrapper(device=\"cpu\")\n",
    "\n",
    "if not COND_STAGE_MODEL_OV_PATH.exists():\n",
    "    convert(\n",
    "        cond_stage_model,\n",
    "        COND_STAGE_MODEL_OV_PATH,\n",
    "        example_input=torch.ones([1, 77], dtype=torch.long),\n",
    "    )\n",
    "\n",
    "del cond_stage_model\n",
    "gc.collect();"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "63b361937c948711",
   "metadata": {},
   "source": [
    "### Convert CLIP image encoder\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "`FrozenOpenCLIPImageEmbedderV2` model accepts images of various resolutions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6aa0a3f-4093-44c7-b7b0-ca830abc4826",
   "metadata": {},
   "outputs": [],
   "source": [
    "EMBEDDER_OV_PATH = MODEL_DIR / \"embedder_ir.xml\"\n",
    "\n",
    "\n",
    "dummy_input = torch.rand([1, 3, 767, 767], dtype=torch.float32)\n",
    "\n",
    "model.embedder.model.visual.input_patchnorm = None  # fix error: visual model has not  attribute 'input_patchnorm'\n",
    "if not EMBEDDER_OV_PATH.exists():\n",
    "    convert(model.embedder, EMBEDDER_OV_PATH, example_input=dummy_input, input_shape=[1, 3, -1, -1])\n",
    "\n",
    "\n",
    "del model.embedder\n",
    "gc.collect();"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "eef65d17fec62fa",
   "metadata": {},
   "source": [
    "### Convert AE encoder\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41434d51-07da-4688-b0af-be6271fef54f",
   "metadata": {},
   "outputs": [],
   "source": [
    "ENCODER_FIRST_STAGE_OV_PATH = MODEL_DIR / \"encoder_first_stage_ir.xml\"\n",
    "\n",
    "\n",
    "dummy_input = torch.rand([1, 3, 256, 256], dtype=torch.float32)\n",
    "\n",
    "if not ENCODER_FIRST_STAGE_OV_PATH.exists():\n",
    "    convert(\n",
    "        model.first_stage_model.encoder,\n",
    "        ENCODER_FIRST_STAGE_OV_PATH,\n",
    "        example_input=dummy_input,\n",
    "    )\n",
    "\n",
    "del model.first_stage_model.encoder\n",
    "gc.collect();"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7ec5ee02317d8e77",
   "metadata": {},
   "source": [
    "### Convert Diffusion U-Net model\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca501d47-3f65-4d58-804d-d8ed20a761af",
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_OV_PATH = MODEL_DIR / \"model_ir.xml\"\n",
    "\n",
    "\n",
    "class ModelWrapper(torch.nn.Module):\n",
    "    def __init__(self, diffusion_model):\n",
    "        super().__init__()\n",
    "        self.diffusion_model = diffusion_model\n",
    "\n",
    "    def forward(self, xc, t, context=None, fs=None, temporal_length=None):\n",
    "        outputs = self.diffusion_model(xc, t, context=context, fs=fs, temporal_length=temporal_length)\n",
    "        return outputs\n",
    "\n",
    "\n",
    "if not MODEL_OV_PATH.exists():\n",
    "    convert(\n",
    "        ModelWrapper(model.model.diffusion_model),\n",
    "        MODEL_OV_PATH,\n",
    "        example_input={\n",
    "            \"xc\": torch.rand([1, 8, 16, 32, 32], dtype=torch.float32),\n",
    "            \"t\": torch.tensor([1]),\n",
    "            \"context\": torch.rand([1, 333, 1024], dtype=torch.float32),\n",
    "            \"fs\": torch.tensor([3]),\n",
    "            \"temporal_length\": torch.tensor([16]),\n",
    "        },\n",
    "    )\n",
    "\n",
    "out_channels = model.model.diffusion_model.out_channels\n",
    "del model.model.diffusion_model\n",
    "gc.collect();"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8d5430af-12b4-4a15-bb7c-c9300f824431",
   "metadata": {},
   "source": [
    "### Convert AE decoder\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "`Decoder` receives a `bfloat16` tensor. numpy doesn't support this type. To avoid problems with the conversion lets replace `decode` method to convert bfloat16 to float32."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "db8394e7-d60b-4dbd-a94a-65e4f21a959b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import types\n",
    "\n",
    "\n",
    "def decode(self, z, **kwargs):\n",
    "    z = self.post_quant_conv(z)\n",
    "    z = z.float()\n",
    "    dec = self.decoder(z)\n",
    "    return dec\n",
    "\n",
    "\n",
    "model.first_stage_model.decode = types.MethodType(decode, model.first_stage_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7c6879b-8b51-4d76-81e1-669378c7c4e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "DECODER_FIRST_STAGE_OV_PATH = MODEL_DIR / \"decoder_first_stage_ir.xml\"\n",
    "\n",
    "\n",
    "dummy_input = torch.rand([16, 4, 32, 32], dtype=torch.float32)\n",
    "\n",
    "if not DECODER_FIRST_STAGE_OV_PATH.exists():\n",
    "    convert(\n",
    "        model.first_stage_model.decoder,\n",
    "        DECODER_FIRST_STAGE_OV_PATH,\n",
    "        example_input=dummy_input,\n",
    "    )\n",
    "\n",
    "del model.first_stage_model.decoder\n",
    "gc.collect();"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "51ff6eb8-dd58-4820-aae3-85c0b4e487a8",
   "metadata": {},
   "source": [
    "## Compiling models\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Select device from dropdown list for running inference using OpenVINO."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8f052ebf-dabe-4161-bee7-4a9d55b9b69a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "53e6e92027184f76b0d73dff67c59464",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from notebook_utils import device_widget\n",
    "\n",
    "core = ov.Core()\n",
    "device = device_widget()\n",
    "\n",
    "device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "410e05b4-11ed-4d85-ab8d-5d1557c4c36e",
   "metadata": {},
   "outputs": [],
   "source": [
    "cond_stage_model = core.read_model(COND_STAGE_MODEL_OV_PATH)\n",
    "encoder_first_stage = core.read_model(ENCODER_FIRST_STAGE_OV_PATH)\n",
    "embedder = core.read_model(EMBEDDER_OV_PATH)\n",
    "model_ov = core.read_model(MODEL_OV_PATH)\n",
    "decoder_first_stage = core.read_model(DECODER_FIRST_STAGE_OV_PATH)\n",
    "\n",
    "compiled_cond_stage_model = core.compile_model(cond_stage_model, device.value)\n",
    "compiled_encode_first_stage = core.compile_model(encoder_first_stage, device.value)\n",
    "compiled_embedder = core.compile_model(embedder, device.value)\n",
    "compiled_model = core.compile_model(model_ov, device.value)\n",
    "compiled_decoder_first_stage = core.compile_model(decoder_first_stage, device.value)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "11f2c95b-e872-458b-a6f8-448f8124ffe6",
   "metadata": {},
   "source": [
    "## Building the pipeline\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Let's create callable wrapper classes for compiled models to allow interaction with original pipelines. Note that all of wrapper classes return `torch.Tensor`s instead of `np.array`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "9d50a153-f71d-4364-9114-14bc25e239d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Any\n",
    "import open_clip\n",
    "\n",
    "\n",
    "class CondStageModelWrapper(torch.nn.Module):\n",
    "    def __init__(self, cond_stage_model):\n",
    "        super().__init__()\n",
    "        self.cond_stage_model = cond_stage_model\n",
    "\n",
    "    def encode(self, tokens):\n",
    "        if isinstance(tokens, list):\n",
    "            tokens = open_clip.tokenize(tokens[0])\n",
    "        outs = self.cond_stage_model(tokens)[0]\n",
    "\n",
    "        return torch.from_numpy(outs)\n",
    "\n",
    "\n",
    "class EncoderFirstStageModelWrapper(torch.nn.Module):\n",
    "    def __init__(self, encode_first_stage):\n",
    "        super().__init__()\n",
    "        self.encode_first_stage = encode_first_stage\n",
    "\n",
    "    def forward(self, x):\n",
    "        outs = self.encode_first_stage(x)[0]\n",
    "\n",
    "        return torch.from_numpy(outs)\n",
    "\n",
    "    def __call__(self, *args: Any, **kwargs: Any) -> Any:\n",
    "        return self.forward(*args, **kwargs)\n",
    "\n",
    "\n",
    "class EmbedderWrapper(torch.nn.Module):\n",
    "    def __init__(self, embedder):\n",
    "        super().__init__()\n",
    "        self.embedder = embedder\n",
    "\n",
    "    def forward(self, x):\n",
    "        outs = self.embedder(x)[0]\n",
    "\n",
    "        return torch.from_numpy(outs)\n",
    "\n",
    "    def __call__(self, *args: Any, **kwargs: Any) -> Any:\n",
    "        return self.forward(*args, **kwargs)\n",
    "\n",
    "\n",
    "class CModelWrapper(torch.nn.Module):\n",
    "    def __init__(self, diffusion_model, out_channels):\n",
    "        super().__init__()\n",
    "        self.diffusion_model = diffusion_model\n",
    "        self.out_channels = out_channels\n",
    "\n",
    "    def forward(self, xc, t, context, fs, temporal_length):\n",
    "        inputs = {\n",
    "            \"xc\": xc,\n",
    "            \"t\": t,\n",
    "            \"context\": context,\n",
    "            \"fs\": fs,\n",
    "        }\n",
    "        outs = self.diffusion_model(inputs)[0]\n",
    "\n",
    "        return torch.from_numpy(outs)\n",
    "\n",
    "    def __call__(self, *args: Any, **kwargs: Any) -> Any:\n",
    "        return self.forward(*args, **kwargs)\n",
    "\n",
    "\n",
    "class DecoderFirstStageModelWrapper(torch.nn.Module):\n",
    "    def __init__(self, decoder_first_stage):\n",
    "        super().__init__()\n",
    "        self.decoder_first_stage = decoder_first_stage\n",
    "\n",
    "    def forward(self, x):\n",
    "        x.float()\n",
    "        outs = self.decoder_first_stage(x)[0]\n",
    "\n",
    "        return torch.from_numpy(outs)\n",
    "\n",
    "    def __call__(self, *args: Any, **kwargs: Any) -> Any:\n",
    "        return self.forward(*args, **kwargs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1178a847-eb14-419b-815e-c47628aa6868",
   "metadata": {},
   "source": [
    "And insert wrappers instances in the pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f45d5dc3-6d17-408e-826d-b8525f461e44",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.cond_stage_model = CondStageModelWrapper(compiled_cond_stage_model)\n",
    "model.first_stage_model.encoder = EncoderFirstStageModelWrapper(compiled_encode_first_stage)\n",
    "model.embedder = EmbedderWrapper(compiled_embedder)\n",
    "model.model.diffusion_model = CModelWrapper(compiled_model, out_channels)\n",
    "model.first_stage_model.decoder = DecoderFirstStageModelWrapper(compiled_decoder_first_stage)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "953f6d14",
   "metadata": {},
   "source": [
    "## Run OpenVINO pipeline inference\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9e7729a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from einops import repeat, rearrange\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "\n",
    "transform = transforms.Compose(\n",
    "    [\n",
    "        transforms.Resize(min((256, 256))),\n",
    "        transforms.CenterCrop((256, 256)),\n",
    "    ]\n",
    ")\n",
    "\n",
    "\n",
    "def get_latent_z(model, videos):\n",
    "    b, c, t, h, w = videos.shape\n",
    "    x = rearrange(videos, \"b c t h w -> (b t) c h w\")\n",
    "    z = model.encode_first_stage(x)\n",
    "    z = rearrange(z, \"(b t) c h w -> b c t h w\", b=b, t=t)\n",
    "    return z\n",
    "\n",
    "\n",
    "def process_input(model, prompt, image, transform=transform, fs=3):\n",
    "    text_emb = model.get_learned_conditioning([prompt])\n",
    "\n",
    "    # img cond\n",
    "    img_tensor = torch.from_numpy(image).permute(2, 0, 1).float().to(model.device)\n",
    "    img_tensor = (img_tensor / 255.0 - 0.5) * 2\n",
    "\n",
    "    image_tensor_resized = transform(img_tensor)  # 3,h,w\n",
    "    videos = image_tensor_resized.unsqueeze(0)  # bchw\n",
    "\n",
    "    z = get_latent_z(model, videos.unsqueeze(2))  # bc,1,hw\n",
    "    frames = model.temporal_length\n",
    "    img_tensor_repeat = repeat(z, \"b c t h w -> b c (repeat t) h w\", repeat=frames)\n",
    "\n",
    "    cond_images = model.embedder(img_tensor.unsqueeze(0))  # blc\n",
    "    img_emb = model.image_proj_model(cond_images)\n",
    "    imtext_cond = torch.cat([text_emb, img_emb], dim=1)\n",
    "\n",
    "    fs = torch.tensor([fs], dtype=torch.long, device=model.device)\n",
    "    cond = {\"c_crossattn\": [imtext_cond], \"fs\": fs, \"c_concat\": [img_tensor_repeat]}\n",
    "    return cond"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "52b6ff5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "from lvdm.models.samplers.ddim import DDIMSampler\n",
    "from pytorch_lightning import seed_everything\n",
    "import torchvision\n",
    "\n",
    "\n",
    "def register_buffer(self, name, attr):\n",
    "    if isinstance(attr, torch.Tensor):\n",
    "        if attr.device != torch.device(\"cpu\"):\n",
    "            attr = attr.to(torch.device(\"cpu\"))\n",
    "    setattr(self, name, attr)\n",
    "\n",
    "\n",
    "def batch_ddim_sampling(model, cond, noise_shape, n_samples=1, ddim_steps=50, ddim_eta=1.0, cfg_scale=1.0, temporal_cfg_scale=None, **kwargs):\n",
    "    ddim_sampler = DDIMSampler(model)\n",
    "    uncond_type = model.uncond_type\n",
    "    batch_size = noise_shape[0]\n",
    "    fs = cond[\"fs\"]\n",
    "    del cond[\"fs\"]\n",
    "    if noise_shape[-1] == 32:\n",
    "        timestep_spacing = \"uniform\"\n",
    "        guidance_rescale = 0.0\n",
    "    else:\n",
    "        timestep_spacing = \"uniform_trailing\"\n",
    "        guidance_rescale = 0.7\n",
    "    # construct unconditional guidance\n",
    "    if cfg_scale != 1.0:\n",
    "        if uncond_type == \"empty_seq\":\n",
    "            prompts = batch_size * [\"\"]\n",
    "            # prompts = N * T * [\"\"]  ## if is_imgbatch=True\n",
    "            uc_emb = model.get_learned_conditioning(prompts)\n",
    "        elif uncond_type == \"zero_embed\":\n",
    "            c_emb = cond[\"c_crossattn\"][0] if isinstance(cond, dict) else cond\n",
    "            uc_emb = torch.zeros_like(c_emb)\n",
    "\n",
    "        # process image embedding token\n",
    "        if hasattr(model, \"embedder\"):\n",
    "            uc_img = torch.zeros(noise_shape[0], 3, 224, 224).to(model.device)\n",
    "            ## img: b c h w >> b l c\n",
    "            uc_img = model.embedder(uc_img)\n",
    "            uc_img = model.image_proj_model(uc_img)\n",
    "            uc_emb = torch.cat([uc_emb, uc_img], dim=1)\n",
    "\n",
    "        if isinstance(cond, dict):\n",
    "            uc = {key: cond[key] for key in cond.keys()}\n",
    "            uc.update({\"c_crossattn\": [uc_emb]})\n",
    "        else:\n",
    "            uc = uc_emb\n",
    "    else:\n",
    "        uc = None\n",
    "\n",
    "    x_T = None\n",
    "    batch_variants = []\n",
    "\n",
    "    for _ in range(n_samples):\n",
    "        if ddim_sampler is not None:\n",
    "            kwargs.update({\"clean_cond\": True})\n",
    "            samples, _ = ddim_sampler.sample(\n",
    "                S=ddim_steps,\n",
    "                conditioning=cond,\n",
    "                batch_size=noise_shape[0],\n",
    "                shape=noise_shape[1:],\n",
    "                verbose=False,\n",
    "                unconditional_guidance_scale=cfg_scale,\n",
    "                unconditional_conditioning=uc,\n",
    "                eta=ddim_eta,\n",
    "                temporal_length=noise_shape[2],\n",
    "                conditional_guidance_scale_temporal=temporal_cfg_scale,\n",
    "                x_T=x_T,\n",
    "                fs=fs,\n",
    "                timestep_spacing=timestep_spacing,\n",
    "                guidance_rescale=guidance_rescale,\n",
    "                **kwargs,\n",
    "            )\n",
    "        # reconstruct from latent to pixel space\n",
    "        batch_images = model.decode_first_stage(samples)\n",
    "        batch_variants.append(batch_images)\n",
    "    # batch, <samples>, c, t, h, w\n",
    "    batch_variants = torch.stack(batch_variants, dim=1)\n",
    "    return batch_variants\n",
    "\n",
    "\n",
    "# monkey patching to replace the original method 'register_buffer' that uses CUDA\n",
    "DDIMSampler.register_buffer = types.MethodType(register_buffer, DDIMSampler)\n",
    "\n",
    "\n",
    "def save_videos(batch_tensors, savedir, filenames, fps=10):\n",
    "    # b,samples,c,t,h,w\n",
    "    n_samples = batch_tensors.shape[1]\n",
    "    for idx, vid_tensor in enumerate(batch_tensors):\n",
    "        video = vid_tensor.detach().cpu()\n",
    "        video = torch.clamp(video.float(), -1.0, 1.0)\n",
    "        video = video.permute(2, 0, 1, 3, 4)  # t,n,c,h,w\n",
    "        frame_grids = [torchvision.utils.make_grid(framesheet, nrow=int(n_samples)) for framesheet in video]  # [3, 1*h, n*w]\n",
    "        grid = torch.stack(frame_grids, dim=0)  # stack in temporal dim [t, 3, n*h, w]\n",
    "        grid = (grid + 1.0) / 2.0\n",
    "        grid = (grid * 255).to(torch.uint8).permute(0, 2, 3, 1)\n",
    "        savepath = os.path.join(savedir, f\"{filenames[idx]}.mp4\")\n",
    "        torchvision.io.write_video(savepath, grid, fps=fps, video_codec=\"h264\", options={\"crf\": \"10\"})\n",
    "\n",
    "\n",
    "def get_image(image, prompt, steps=5, cfg_scale=7.5, eta=1.0, fs=3, seed=123, model=model, result_dir=\"results\"):\n",
    "    if not os.path.exists(result_dir):\n",
    "        os.mkdir(result_dir)\n",
    "\n",
    "    seed_everything(seed)\n",
    "\n",
    "    # torch.cuda.empty_cache()\n",
    "    print(\"start:\", prompt, time.strftime(\"%Y-%m-%d %H:%M:%S\", time.localtime(time.time())))\n",
    "    start = time.time()\n",
    "    if steps > 60:\n",
    "        steps = 60\n",
    "    model = model.cpu()\n",
    "    batch_size = 1\n",
    "    channels = model.model.diffusion_model.out_channels\n",
    "    frames = model.temporal_length\n",
    "    h, w = 256 // 8, 256 // 8\n",
    "    noise_shape = [batch_size, channels, frames, h, w]\n",
    "\n",
    "    # text cond\n",
    "    with torch.no_grad(), torch.cpu.amp.autocast():\n",
    "        cond = process_input(model, prompt, image, transform, fs=3)\n",
    "\n",
    "        ## inference\n",
    "        batch_samples = batch_ddim_sampling(model, cond, noise_shape, n_samples=1, ddim_steps=steps, ddim_eta=eta, cfg_scale=cfg_scale)\n",
    "        ## b,samples,c,t,h,w\n",
    "        prompt_str = prompt.replace(\"/\", \"_slash_\") if \"/\" in prompt else prompt\n",
    "        prompt_str = prompt_str.replace(\" \", \"_\") if \" \" in prompt else prompt_str\n",
    "        prompt_str = prompt_str[:40]\n",
    "        if len(prompt_str) == 0:\n",
    "            prompt_str = \"empty_prompt\"\n",
    "\n",
    "    save_videos(batch_samples, result_dir, filenames=[prompt_str], fps=8)\n",
    "    print(f\"Saved in {prompt_str}.mp4. Time used: {(time.time() - start):.2f} seconds\")\n",
    "\n",
    "    return os.path.join(result_dir, f\"{prompt_str}.mp4\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9a0137a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Seed set to 234\n",
      "/tmp/ipykernel_971108/2451984876.py:25: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:206.)\n",
      "  img_tensor = torch.from_numpy(image).permute(2, 0, 1).float().to(model.device)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start: man fishing in a boat at sunset 2024-08-06 13:54:24\n",
      "Saved in man_fishing_in_a_boat_at_sunset.mp4. Time used: 164.28 seconds\n"
     ]
    }
   ],
   "source": [
    "image_path = \"DynamiCrafter/prompts/256/art.png\"\n",
    "prompt = \"man fishing in a boat at sunset\"\n",
    "seed = 234\n",
    "image = Image.open(image_path)\n",
    "image = np.asarray(image)\n",
    "result_dir = \"results\"\n",
    "video_path = get_image(image, prompt, steps=20, seed=seed, model=model, result_dir=result_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "3734113f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <video alt=\"video\" controls>\n",
       "        <source src=\"results/man_fishing_in_a_boat_at_sunset.mp4\" type=\"video/mp4\">\n",
       "    </video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "HTML(\n",
    "    f\"\"\"\n",
    "    <video alt=\"video\" controls>\n",
    "        <source src=\"{video_path}\" type=\"video/mp4\">\n",
    "    </video>\n",
    "\"\"\"\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0e8548d7",
   "metadata": {},
   "source": [
    "## Quantization\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "[NNCF](https://github.com/openvinotoolkit/nncf/) enables post-training quantization by adding quantization layers into model graph and then using a subset of the training dataset to initialize the parameters of these additional quantization layers. Quantized operations are executed in `INT8` instead of `FP32`/`FP16` making model inference faster.\n",
    "\n",
    "According to `DynamiCrafter` structure, denoising UNet model is used in the cycle repeating inference on each diffusion step, while other parts of pipeline take part only once. Now we will show you how to optimize pipeline using [NNCF](https://github.com/openvinotoolkit/nncf/) to reduce memory and computation cost.\n",
    "\n",
    "Please select below whether you would like to run quantization to improve model inference speed.\n",
    "\n",
    "> **NOTE**: Quantization is time and memory consuming operation. Running quantization code below may take some time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f991e3c1",
   "metadata": {
    "test_replace": {
     "to_quantize = quantization_widget()": "to_quantize = quantization_widget(False)"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3d982aa54c5e4fdca03268dd91de8da4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Checkbox(value=True, description='Quantization')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from notebook_utils import quantization_widget\n",
    "\n",
    "to_quantize = quantization_widget()\n",
    "\n",
    "to_quantize"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "03c3dfb5",
   "metadata": {},
   "source": [
    "Let's load `skip magic` extension to skip quantization if `to_quantize` is not selected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "61d199d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fetch `skip_kernel_extension` module\n",
    "import requests\n",
    "\n",
    "r = requests.get(\n",
    "    url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/skip_kernel_extension.py\",\n",
    ")\n",
    "open(\"skip_kernel_extension.py\", \"w\").write(r.text)\n",
    "\n",
    "int8_model = None\n",
    "MODEL_INT8_OV_PATH = MODEL_DIR / \"model_ir_int8.xml\"\n",
    "COND_STAGE_MODEL_INT8_OV_PATH = MODEL_DIR / \"cond_stage_model_int8.xml\"\n",
    "DECODER_FIRST_STAGE_INT8_OV_PATH = MODEL_DIR / \"decoder_first_stage_ir_int8.xml\"\n",
    "ENCODER_FIRST_STAGE_INT8_OV_PATH = MODEL_DIR / \"encoder_first_stage_ir_int8.xml\"\n",
    "EMBEDDER_INT8_OV_PATH = MODEL_DIR / \"embedder_ir_int8.xml\"\n",
    "\n",
    "%load_ext skip_kernel_extension"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2c5ff698",
   "metadata": {},
   "source": [
    "### Prepare calibration dataset\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "We use a portion of [`jovianzm/Pexels-400k`](https://huggingface.co/datasets/jovianzm/Pexels-400k) dataset from Hugging Face as calibration data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c1b7bb27",
   "metadata": {},
   "outputs": [],
   "source": [
    "from io import BytesIO\n",
    "\n",
    "\n",
    "def download_image(url):\n",
    "    try:\n",
    "        response = requests.get(url)\n",
    "        response.raise_for_status()\n",
    "        img = Image.open(BytesIO(response.content))\n",
    "        # Convert the image to a NumPy array\n",
    "        img_array = np.array(img)\n",
    "        return img_array\n",
    "    except Exception as err:\n",
    "        print(f\"Error occurred: {err}\")\n",
    "        return None"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fb54119d",
   "metadata": {},
   "source": [
    "To collect intermediate model inputs for calibration we should customize `CompiledModel`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "7c141d3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "import datasets\n",
    "from tqdm.notebook import tqdm\n",
    "import pickle\n",
    "\n",
    "\n",
    "class CompiledModelDecorator(ov.CompiledModel):\n",
    "    def __init__(self, compiled_model, keep_prob, data_cache = None):\n",
    "        super().__init__(compiled_model)\n",
    "        self.data_cache = data_cache if data_cache else []\n",
    "        self.keep_prob = np.clip(keep_prob, 0, 1)\n",
    "\n",
    "    def __call__(self, *args, **kwargs):\n",
    "        if np.random.rand() <= self.keep_prob:\n",
    "            self.data_cache.append(*args)\n",
    "        return super().__call__(*args, **kwargs)\n",
    "\n",
    "def collect_calibration_data(model, subset_size):\n",
    "    calibration_dataset_filepath = Path(\"calibration_data\")/f\"{subset_size}.pkl\"\n",
    "    calibration_dataset_filepath.parent.mkdir(exist_ok=True, parents=True)\n",
    "    if not calibration_dataset_filepath.exists():\n",
    "        original_diffusion_model = model.model.diffusion_model.diffusion_model\n",
    "        modified_model = CompiledModelDecorator(original_diffusion_model, keep_prob=1)\n",
    "        model.model.diffusion_model = CModelWrapper(modified_model, model.model.diffusion_model.out_channels)\n",
    "    \n",
    "        dataset = datasets.load_dataset(\"google-research-datasets/conceptual_captions\", trust_remote_code=True, split=\"train\", streaming=True).shuffle(seed=42).take(subset_size)\n",
    "    \n",
    "        pbar = tqdm(total=subset_size)\n",
    "        channels = model.model.diffusion_model.out_channels\n",
    "        frames = model.temporal_length\n",
    "        h, w = 256 // 8, 256 // 8\n",
    "        noise_shape = [1, channels, frames, h, w]\n",
    "        for batch in dataset:\n",
    "            prompt = batch[\"caption\"]\n",
    "            image_path = batch[\"image_url\"]\n",
    "            image = download_image(image_path)\n",
    "            if image is None:\n",
    "                continue\n",
    "    \n",
    "            cond = process_input(model, prompt, image)\n",
    "            batch_ddim_sampling(model, cond, noise_shape, n_samples=1, ddim_steps=20, ddim_eta=1.0, cfg_scale=7.5)\n",
    "    \n",
    "            collected_subset_size = len(model.model.diffusion_model.diffusion_model.data_cache)\n",
    "            if collected_subset_size >= subset_size:\n",
    "                pbar.update(subset_size - pbar.n)\n",
    "                break\n",
    "            pbar.update(collected_subset_size - pbar.n)\n",
    "    \n",
    "        calibration_dataset = model.model.diffusion_model.diffusion_model.data_cache[:subset_size]\n",
    "        model.model.diffusion_model.diffusion_model = original_diffusion_model\n",
    "        with open(calibration_dataset_filepath, 'wb') as f:\n",
    "            pickle.dump(calibration_dataset, f)\n",
    "    with open(calibration_dataset_filepath, 'rb') as f:\n",
    "        calibration_dataset = pickle.load(f)\n",
    "    return calibration_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "1213b8cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "32f25d7199e941228353d2d1b2228435",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/300 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "\n",
    "if not MODEL_INT8_OV_PATH.exists():\n",
    "    subset_size = 300\n",
    "    calibration_data = collect_calibration_data(model, subset_size=subset_size)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "12c23abd",
   "metadata": {},
   "source": [
    "### Run Quantization\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Quantization of the first and last `Convolution` layers impacts the generation results. We recommend using `IgnoredScope` to keep accuracy sensitive layers in FP16 precision. `FastBiasCorrection` algorithm is disabled due to minimal accuracy improvement in SD models and increased quantization time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "37af0dd3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, openvino\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "55e253254be646f4a514825feaa56101",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dbbe05b882a84f9485e2ca9eedc2cf28",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:2 ignored nodes were found by name in the NNCFGraph\n",
      "INFO:nncf:269 ignored nodes were found by name in the NNCFGraph\n",
      "INFO:nncf:Not adding activation input quantizer for operation: 69 __module.diffusion_model.input_blocks.0.0/aten::_convolution/Convolution\n",
      "165 __module.diffusion_model.input_blocks.0.0/aten::_convolution/Add\n",
      "\n",
      "INFO:nncf:Not adding activation input quantizer for operation: 4107 __module.diffusion_model.out.2/aten::_convolution/Convolution\n",
      "4411 __module.diffusion_model.out.2/aten::_convolution/Add\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6a02a675ca0642d5be47e208b7dd4a80",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "import nncf\n",
    "\n",
    "\n",
    "if MODEL_INT8_OV_PATH.exists():\n",
    "    print(\"Model already quantized\")\n",
    "else:\n",
    "    ov_model_ir = core.read_model(MODEL_OV_PATH)\n",
    "    quantized_model = nncf.quantize(\n",
    "        model=ov_model_ir,\n",
    "        subset_size=subset_size,\n",
    "        calibration_dataset=nncf.Dataset(calibration_data),\n",
    "        model_type=nncf.ModelType.TRANSFORMER,\n",
    "        ignored_scope=nncf.IgnoredScope(names=[\n",
    "            \"__module.diffusion_model.input_blocks.0.0/aten::_convolution/Convolution\",\n",
    "            \"__module.diffusion_model.out.2/aten::_convolution/Convolution\",\n",
    "        ]),\n",
    "        advanced_parameters=nncf.AdvancedQuantizationParameters(disable_bias_correction=True)\n",
    "    )\n",
    "    ov.save_model(quantized_model, MODEL_INT8_OV_PATH)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "011835c3",
   "metadata": {},
   "source": [
    "### Run Weights Compression\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Quantizing of the remaining components of the pipeline does not significantly improve inference performance but can lead to a substantial degradation of accuracy. The weight compression will be applied to footprint reduction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "fe98145f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:Statistics of the bitwidth distribution:\n",
      "┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑\n",
      "│   Num bits (N) │ % all parameters (layers)   │ % ratio-defining parameters (layers)   │\n",
      "┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥\n",
      "│              8 │ 100% (97 / 97)              │ 100% (97 / 97)                         │\n",
      "┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3e5dff48f35b48dfae4d286dbd82276d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:Statistics of the bitwidth distribution:\n",
      "┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑\n",
      "│   Num bits (N) │ % all parameters (layers)   │ % ratio-defining parameters (layers)   │\n",
      "┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥\n",
      "│              8 │ 100% (39 / 39)              │ 100% (39 / 39)                         │\n",
      "┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d72f11b0bbf947a8b52b00e88f603585",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:Statistics of the bitwidth distribution:\n",
      "┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑\n",
      "│   Num bits (N) │ % all parameters (layers)   │ % ratio-defining parameters (layers)   │\n",
      "┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥\n",
      "│              8 │ 100% (31 / 31)              │ 100% (31 / 31)                         │\n",
      "┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1d5c4ecfc3024cc389f9264ea8b923b6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:Statistics of the bitwidth distribution:\n",
      "┍━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑\n",
      "│   Num bits (N) │ % all parameters (layers)   │ % ratio-defining parameters (layers)   │\n",
      "┝━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥\n",
      "│              8 │ 100% (129 / 129)            │ 100% (129 / 129)                       │\n",
      "┕━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c54b709a68094c36928e433565cb2767",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "def compress_model_weights(fp_model_path, int8_model_path):\n",
    "    if not int8_model_path.exists():\n",
    "        model = core.read_model(fp_model_path)\n",
    "        compressed_model = nncf.compress_weights(model)\n",
    "        ov.save_model(compressed_model, int8_model_path)\n",
    "\n",
    "\n",
    "compress_model_weights(COND_STAGE_MODEL_OV_PATH, COND_STAGE_MODEL_INT8_OV_PATH)\n",
    "compress_model_weights(DECODER_FIRST_STAGE_OV_PATH, DECODER_FIRST_STAGE_INT8_OV_PATH)\n",
    "compress_model_weights(ENCODER_FIRST_STAGE_OV_PATH, ENCODER_FIRST_STAGE_INT8_OV_PATH)\n",
    "compress_model_weights(EMBEDDER_OV_PATH, EMBEDDER_INT8_OV_PATH)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6c8d7527",
   "metadata": {},
   "source": [
    "Let's run the optimized pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "d311de3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "compiled_cond_stage_model = core.compile_model(core.read_model(COND_STAGE_MODEL_INT8_OV_PATH), device.value)\n",
    "compiled_encode_first_stage = core.compile_model(core.read_model(ENCODER_FIRST_STAGE_INT8_OV_PATH), device.value)\n",
    "compiled_embedder = core.compile_model(core.read_model(EMBEDDER_INT8_OV_PATH), device.value)\n",
    "compiled_model = core.compile_model(core.read_model(MODEL_INT8_OV_PATH), device.value)\n",
    "compiled_decoder_first_stage = core.compile_model(core.read_model(DECODER_FIRST_STAGE_INT8_OV_PATH), device.value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "a0ce59ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AE working on z of shape (1, 4, 32, 32) = 4096 dimensions.\n",
      ">>> model checkpoint loaded.\n"
     ]
    }
   ],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "int8_model = download_model()\n",
    "int8_model.first_stage_model.decode = types.MethodType(decode, int8_model.first_stage_model)\n",
    "int8_model.embedder.model.visual.input_patchnorm = None  # fix error: visual model has not  attribute 'input_patchnorm'\n",
    "\n",
    "int8_model.cond_stage_model = CondStageModelWrapper(compiled_cond_stage_model)\n",
    "int8_model.first_stage_model.encoder = EncoderFirstStageModelWrapper(compiled_encode_first_stage)\n",
    "int8_model.embedder = EmbedderWrapper(compiled_embedder)\n",
    "int8_model.model.diffusion_model = CModelWrapper(compiled_model, out_channels)\n",
    "int8_model.first_stage_model.decoder = DecoderFirstStageModelWrapper(compiled_decoder_first_stage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e77da42",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Seed set to 234\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start: man fishing in a boat at sunset 2024-08-06 15:09:26\n",
      "Saved in man_fishing_in_a_boat_at_sunset.mp4. Time used: 81.47 seconds\n"
     ]
    }
   ],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "image_path = \"DynamiCrafter/prompts/256/art.png\"\n",
    "prompt = \"man fishing in a boat at sunset\"\n",
    "seed = 234\n",
    "image = Image.open(image_path)\n",
    "image = np.asarray(image)\n",
    "\n",
    "result_dir = \"results_int8\"\n",
    "video_path = get_image(image, prompt, steps=20, seed=seed, model=int8_model, result_dir=result_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "d8a817f6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <video alt=\"video\" controls>\n",
       "        <source src=results_int8/man_fishing_in_a_boat_at_sunset.mp4 type=\"video/mp4\">\n",
       "    </video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "display(HTML(f\"\"\"\n",
    "    <video alt=\"video\" controls>\n",
    "        <source src={video_path} type=\"video/mp4\">\n",
    "    </video>\n",
    "\"\"\"))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "81410f5a",
   "metadata": {},
   "source": [
    "### Compare model file sizes\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "f2134675",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cond_stage_model compression rate: 3.977\n",
      "decoder_first_stage_ir compression rate: 3.987\n",
      "encoder_first_stage_ir compression rate: 3.986\n",
      "embedder_ir compression rate: 3.977\n",
      "model_ir compression rate: 3.981\n"
     ]
    }
   ],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "fp32_model_paths = [COND_STAGE_MODEL_OV_PATH, DECODER_FIRST_STAGE_OV_PATH, ENCODER_FIRST_STAGE_OV_PATH, EMBEDDER_OV_PATH, MODEL_OV_PATH]\n",
    "int8_model_paths = [COND_STAGE_MODEL_INT8_OV_PATH, DECODER_FIRST_STAGE_INT8_OV_PATH, ENCODER_FIRST_STAGE_INT8_OV_PATH, EMBEDDER_INT8_OV_PATH, MODEL_INT8_OV_PATH]\n",
    "\n",
    "for fp16_path, int8_path in zip(fp32_model_paths, int8_model_paths):\n",
    "    fp32_ir_model_size = fp16_path.with_suffix(\".bin\").stat().st_size\n",
    "    int8_model_size = int8_path.with_suffix(\".bin\").stat().st_size\n",
    "    print(f\"{fp16_path.stem} compression rate: {fp32_ir_model_size / int8_model_size:.3f}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fa624ee9",
   "metadata": {},
   "source": [
    "### Compare inference time of the FP32 and INT8 models\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "To measure the inference performance of the `FP32` and `INT8` models, we use median inference time on calibration subset.\n",
    "\n",
    "> **NOTE**: For the most accurate performance estimation, it is recommended to run `benchmark_app` in a terminal/command prompt after closing other applications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "7755a0cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "import time\n",
    "\n",
    "\n",
    "def calculate_inference_time(model, validation_size=3):\n",
    "    calibration_dataset = datasets.load_dataset(\"jovianzm/Pexels-400k\", split=\"train\", streaming=True).take(validation_size)\n",
    "    inference_time = []\n",
    "    channels = model.model.diffusion_model.out_channels\n",
    "    frames = model.temporal_length\n",
    "    h, w = 256 // 8, 256 // 8\n",
    "    noise_shape = [1, channels, frames, h, w]\n",
    "    for batch in calibration_dataset:\n",
    "        prompt = batch[\"title\"]\n",
    "        image_path = batch[\"thumbnail\"]\n",
    "        image = download_image(image_path)\n",
    "        cond = process_input(model, prompt, image, transform, fs=3)\n",
    "\n",
    "        start = time.perf_counter()\n",
    "        _ = batch_ddim_sampling(model, cond, noise_shape, n_samples=1, ddim_steps=20, ddim_eta=1.0, cfg_scale=7.5)\n",
    "        end = time.perf_counter()\n",
    "        delta = end - start\n",
    "        inference_time.append(delta)\n",
    "    return np.median(inference_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "e61b1152",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FP32 latency: 162.304\n",
      "INT8 latency: 79.590\n",
      "Performance speed up: 2.039\n"
     ]
    }
   ],
   "source": [
    "%%skip not $to_quantize.value\n",
    "\n",
    "fp_latency = calculate_inference_time(model)\n",
    "print(f\"FP32 latency: {fp_latency:.3f}\")\n",
    "int8_latency = calculate_inference_time(int8_model)\n",
    "print(f\"INT8 latency: {int8_latency:.3f}\")\n",
    "print(f\"Performance speed up: {fp_latency / int8_latency:.3f}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4417db2b-2f65-407c-a384-ec466f18bca0",
   "metadata": {},
   "source": [
    "## Interactive inference\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Please select below whether you would like to use the quantized models to launch the interactive demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cdf535e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import widgets\n",
    "\n",
    "quantized_models_present = int8_model is not None\n",
    "\n",
    "use_quantized_models = widgets.Checkbox(\n",
    "    value=quantized_models_present,\n",
    "    description=\"Use quantized models\",\n",
    "    disabled=not quantized_models_present,\n",
    ")\n",
    "\n",
    "use_quantized_models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf57f8a8-8cf6-45c5-ae78-02a3ed04fcc8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "\n",
    "demo_model = int8_model if use_quantized_models.value else model\n",
    "get_image_fn = partial(get_image, model=demo_model)\n",
    "\n",
    "if not Path(\"gradio_helper.py\").exists():\n",
    "    r = requests.get(\n",
    "        url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/dynamicrafter-animating-images/gradio_helper.py\"\n",
    "    )\n",
    "    open(\"gradio_helper.py\", \"w\").write(r.text)\n",
    "\n",
    "from gradio_helper import make_demo\n",
    "\n",
    "demo = make_demo(fn=get_image_fn)\n",
    "\n",
    "try:\n",
    "    demo.queue().launch(debug=True)\n",
    "except Exception:\n",
    "    demo.queue().launch(debug=True, share=True)\n",
    "# if you are launching remotely, specify server_name and server_port\n",
    "# demo.launch(server_name='your server name', server_port='server port in int')\n",
    "# Read more in the docs: https://gradio.app/docs/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "openvino_notebooks": {
   "imageUrl": "https://github.com/Doubiiu/DynamiCrafter/blob/main/assets/showcase/guitar0.gif?raw=true",
   "tags": {
    "categories": [
     "Model Demos",
     "AI Trends"
    ],
    "libraries": [],
    "other": [],
    "tasks": [
     "Image-to-Video"
    ]
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}