notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimizing Transformers with Optimum\n",
    "\n",
    "In this session, you will learn how to optimize Hugging Face Transformers models using Optimum. The session will show you how to dynamically quantize and optimize a DistilBERT model using [Hugging Face Optimum](https://huggingface.co/docs/optimum/index) and [ONNX Runtime](https://onnxruntime.ai/). Hugging Face Optimum is an extension of 🤗 Transformers, providing a set of performance optimization tools enabling maximum efficiency to train and run models on targeted hardware.\n",
    "\n",
    "Note: dynamic quantization is currently only supported for CPUs, so we will not be utilizing GPUs / CUDA in this session.\n",
    "\n",
    "By the end of this session, you see how quantization and optimization with Hugging Face Optimum can result in significant increase in model latency while keeping almost 100% of the full-precision model. Furthermore, you’ll see how to easily apply some advanced quantization and optimization techniques shown here so that your models take much less of an accuracy hit than they would otherwise. \n",
    "\n",
    "You will learn how to:\n",
    "1. Setup Development Environment\n",
    "2. Convert a Hugging Face `Transformers` model to ONNX for inference\n",
    "3. Apply graph optimization techniques to the ONNX model\n",
    "4. Apply dynamic quantization using `ORTQuantizer` from Optimum\n",
    "5. Test inference with the quantized model\n",
    "6. Evaluate the performance and speed\n",
    "7. Push the quantized model to the Hub\n",
    "8. Load and run inference with a quantized model from the hub\n",
    "\n",
    "Let's get started! 🚀\n",
    "\n",
    "_This tutorial was created and run on an c6i.xlarge AWS EC2 Instance._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Setup Development Environment\n",
    "\n",
    "Our first step is to install Optimum, along with  Evaluate and some other libraries. Running the following cell will install all the required packages for us including Transformers, PyTorch, and ONNX Runtime utilities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install \"optimum[onnxruntime]==1.5.0\" evaluate[evaluator] sklearn mkl-include mkl --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> If you want to run inference on a GPU, you can install 🤗 Optimum with `pip install optimum[onnxruntime-gpu]`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Convert a Hugging Face `Transformers` model to ONNX for inference\n",
    "\n",
    "Before we can start qunatizing we need to convert our vanilla `transformers` model to the `onnx` format. To do this we will use the new [ORTModelForSequenceClassification](https://huggingface.co/docs/optimum/main/en/onnxruntime/modeling_ort#optimum.onnxruntime.ORTModelForSequenceClassification) class calling the `from_pretrained()` method with the `from_transformers` attribute. The model we are using is the [optimum/distilbert-base-uncased-finetuned-banking77](https://huggingface.co/optimum/distilbert-base-uncased-finetuned-banking77) a fine-tuned DistilBERT model on the Banking77 dataset achieving an Accuracy score of `92.5` and as the feature (task) `text-classification`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7fcdcba4efb846a8a9d26790f6ee2f58",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/5.81k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ff3475d9d3bb44b4a7a5e580f1c07449",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/333 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "00317439669f42da89631556e3fd0669",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a55a157c1e604313a6aff51d1dc51292",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/711k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0767fe7a7080494c8866ee000e93a109",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/112 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b0763d0b6a0c46819f7c6ee4849b375e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/268M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/miniconda3/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py:213: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.\n",
      "  mask, torch.tensor(torch.finfo(scores.dtype).min)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('onnx/tokenizer_config.json',\n",
       " 'onnx/special_tokens_map.json',\n",
       " 'onnx/vocab.txt',\n",
       " 'onnx/added_tokens.json',\n",
       " 'onnx/tokenizer.json')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from optimum.onnxruntime import ORTModelForSequenceClassification\n",
    "from transformers import AutoTokenizer\n",
    "from pathlib import Path\n",
    "\n",
    "\n",
    "model_id=\"optimum/distilbert-base-uncased-finetuned-banking77\"\n",
    "dataset_id=\"banking77\"\n",
    "onnx_path = Path(\"onnx\")\n",
    "\n",
    "# load vanilla transformers and convert to onnx\n",
    "model = ORTModelForSequenceClassification.from_pretrained(model_id, from_transformers=True)\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "\n",
    "# save onnx checkpoint and tokenizer\n",
    "model.save_pretrained(onnx_path)\n",
    "tokenizer.save_pretrained(onnx_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One neat thing about 🤗 Optimum, is that allows you to run ONNX models with the `pipeline()` function from 🤗 Transformers. This means that you get all the pre- and post-processing features for free, without needing to re-implement them for each model! Here's how you can run inference with our vanilla ONNX model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'label': 'lost_or_stolen_card', 'score': 0.9664045572280884}]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "vanilla_clf = pipeline(\"text-classification\", model=model, tokenizer=tokenizer)\n",
    "vanilla_clf(\"Could you assist me in finding my lost card?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to learn more about exporting transformers model check-out [Convert Transformers to ONNX with Hugging Face Optimum](https://www.philschmid.de/convert-transformers-to-onnx) blog post\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Apply graph optimization techniques to the ONNX model\n",
    "\n",
    "Graph optimizations are essentially graph-level transformations, ranging from small graph simplifications and node eliminations to more complex node fusions and layout optimizations. \n",
    "Examples of graph optimizations include:\n",
    "* **Constant folding**: evaluate constant expressions at compile time instead of runtime\n",
    "* **Redundant node elimination**: remove redundant nodes without changing graph structure\n",
    "* **Operator fusion**: merge one node (i.e. operator) into another so they can be executed together\n",
    "\n",
    "\n",
    "![operator fusion](./assets/operator_fusion.png)\n",
    "\n",
    "If you want to learn more about graph optimization you take a look at the [ONNX Runtime documentation](https://onnxruntime.ai/docs/performance/graph-optimizations.html). We are going to first optimize the model and then dynamically quantize to be able to use transformers specific operators such as QAttention for quantization of attention layers.\n",
    "To apply graph optimizations to our ONNX model, we will use the `ORTOptimizer()`. The `ORTOptimizer` makes it with the help of a `OptimizationConfig` easy to optimize. The `OptimizationConfig` is the configuration class handling all the ONNX Runtime optimization parameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from optimum.onnxruntime import ORTOptimizer\n",
    "from optimum.onnxruntime.configuration import OptimizationConfig\n",
    "\n",
    "# create ORTOptimizer and define optimization configuration\n",
    "optimizer = ORTOptimizer.from_pretrained(model)\n",
    "optimization_config = OptimizationConfig(optimization_level=99) # enable all optimizations\n",
    "\n",
    "# apply the optimization configuration to the model\n",
    "optimizer.optimize(\n",
    "    save_dir=onnx_path,\n",
    "    optimization_config=optimization_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To test performance we can use the ORTModelForSequenceClassification class again and provide an additional `file_name` parameter to load our optimized model. _(This also works for models available on the hub)._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'label': 'lost_or_stolen_card', 'score': 0.9664045572280884}]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "# load optimized model\n",
    "model = ORTModelForSequenceClassification.from_pretrained(onnx_path, file_name=\"model_optimized.onnx\")\n",
    "\n",
    "# create optimized pipeline\n",
    "optimized_clf = pipeline(\"text-classification\", model=model, tokenizer=tokenizer)\n",
    "optimized_clf(\"Could you assist me in finding my lost card?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Apply dynamic quantization using `ORTQuantizer` from Optimum\n",
    "\n",
    "After we have optimized our model we can accelerate it even more by quantizing it using the `ORTQuantizer`. The `ORTQuantizer` can be used to apply dynamic quantization to decrease the size of the model size and accelerate latency and inference.\n",
    "\n",
    "_We use the `avx512_vnni` config since the instance is powered by an intel ice-lake CPU supporting avx512._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from optimum.onnxruntime import ORTQuantizer\n",
    "from optimum.onnxruntime.configuration import AutoQuantizationConfig\n",
    "\n",
    "# create ORTQuantizer and define quantization configuration\n",
    "dynamic_quantizer = ORTQuantizer.from_pretrained(model)\n",
    "dqconfig = AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=False)\n",
    "\n",
    "# apply the quantization configuration to the model\n",
    "model_quantized_path = dynamic_quantizer.quantize(\n",
    "    save_dir=onnx_path,\n",
    "    quantization_config=dqconfig,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets quickly check the new model size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model file size: 255.65 MB\n",
      "Quantized Model file size: 162.68 MB\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "# get model file size\n",
    "size = os.path.getsize(onnx_path / \"model_optimized.onnx\")/(1024*1024)\n",
    "quantized_model = os.path.getsize(onnx_path / \"model_optimized_quantized.onnx\")/(1024*1024)\n",
    "\n",
    "print(f\"Model file size: {size:.2f} MB\")\n",
    "print(f\"Quantized Model file size: {quantized_model:.2f} MB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Test inference with the quantized model\n",
    "\n",
    "[Optimum](https://huggingface.co/docs/optimum/main/en/pipelines#optimizing-with-ortoptimizer) has built-in support for [transformers pipelines](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#pipelines). This allows us to leverage the same API that we know from using PyTorch and TensorFlow models.\n",
    "Therefore we can load our quantized model with `ORTModelForSequenceClassification` class and transformers `pipeline`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'label': 'exchange_rate', 'score': 0.9792892336845398}]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from optimum.onnxruntime import ORTModelForSequenceClassification\n",
    "from transformers import pipeline, AutoTokenizer\n",
    "\n",
    "model = ORTModelForSequenceClassification.from_pretrained(onnx_path,file_name=\"model_optimized_quantized.onnx\")\n",
    "tokenizer = AutoTokenizer.from_pretrained(onnx_path)\n",
    "\n",
    "q8_clf = pipeline(\"text-classification\",model=model, tokenizer=tokenizer)\n",
    "\n",
    "q8_clf(\"What is the exchange rate like on this app?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Evaluate the performance and speed\n",
    "\n",
    "We can now leverage the map function of datasets to iterate over the validation set of squad 2 and run prediction for each data point. Therefore we write a evaluate helper method which uses our pipelines and applies some transformation to work with the squad v2 metric.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ea013c3d7fb5460baab887da0a6fabd5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading builder script:   0%|          | 0.00/2.34k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b0d36c33b4e7441fa038783dbeb0f63a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading metadata:   0%|          | 0.00/1.75k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:datasets.builder:Using custom data configuration default\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading and preparing dataset banking77/default (download: 1.03 MiB, generated: 897.51 KiB, post-processed: Unknown size, total: 1.91 MiB) to /home/ubuntu/.cache/huggingface/datasets/banking77/default/1.1.0/aec0289529599d4572d76ab00c8944cb84f88410ad0c9e7da26189d31f62a55b...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e0bf883adada485c848027cc51615a81",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/158k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b3d5d00d5609451c9d39d6bc4f99cf1d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/51.1k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6531ab7c63f641438d04b7aef3c8396a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train split:   0%|          | 0/10003 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "30321c8112be41c594d2a0ae64ceb6e2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating test split:   0%|          | 0/3080 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset banking77 downloaded and prepared to /home/ubuntu/.cache/huggingface/datasets/banking77/default/1.1.0/aec0289529599d4572d76ab00c8944cb84f88410ad0c9e7da26189d31f62a55b. Subsequent calls will reuse this data.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cba4cb3b35004b83ad3cd419aa40a192",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading builder script:   0%|          | 0.00/4.20k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'accuracy': 0.9227272727272727, 'total_time_in_seconds': 19.384229860999994, 'samples_per_second': 158.89204895350477, 'latency_in_seconds': 0.006293581123701297}\n"
     ]
    }
   ],
   "source": [
    "from evaluate import evaluator\n",
    "from datasets import load_dataset \n",
    "\n",
    "eval = evaluator(\"text-classification\")\n",
    "eval_dataset = load_dataset(\"banking77\", split=\"test\")\n",
    "\n",
    "results = eval.compute(\n",
    "    model_or_pipeline=q8_clf,\n",
    "    data=eval_dataset,\n",
    "    metric=\"accuracy\",\n",
    "    input_column=\"text\",\n",
    "    label_column=\"label\",\n",
    "    label_mapping=model.config.label2id,\n",
    "    strategy=\"simple\",\n",
    ")\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vanilla model: 92.5%\n",
      "Quantized model: 92.27%\n",
      "The quantized model achieves 99.75% accuracy of the fp32 model\n"
     ]
    }
   ],
   "source": [
    "print(f\"Vanilla model: 92.5%\")\n",
    "print(f\"Quantized model: {results['accuracy']*100:.2f}%\")\n",
    "print(f\"The quantized model achieves {round(results['accuracy']/0.925,4)*100:.2f}% accuracy of the fp32 model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, now let's test the performance (latency) of our quantized model. We are going to use a payload with a sequence length of 128 for the benchmark. To keep it simple, we are going to use a python loop and calculate the avg,mean & p95 latency for our vanilla model and for the quantized model.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import perf_counter\n",
    "import numpy as np \n",
    "\n",
    "payload=\"Hello my name is Philipp. I am getting in touch with you because i didn't get a response from you. What do I need to do to get my new card which I have requested 2 weeks ago? Please help me and answer this email in the next 7 days. Best regards and have a nice weekend \"*2\n",
    "print(f'Payload sequence length: {len(tokenizer(payload)[\"input_ids\"])}')\n",
    "\n",
    "def measure_latency(pipe):\n",
    "    latencies = []\n",
    "    # warm up\n",
    "    for _ in range(10):\n",
    "        _ = pipe(payload)\n",
    "    # Timed run\n",
    "    for _ in range(300):\n",
    "        start_time = perf_counter()\n",
    "        _ =  pipe(payload)\n",
    "        latency = perf_counter() - start_time\n",
    "        latencies.append(latency)\n",
    "    # Compute run statistics\n",
    "    time_avg_ms = 1000 * np.mean(latencies)\n",
    "    time_std_ms = 1000 * np.std(latencies)\n",
    "    time_p95_ms = 1000 * np.percentile(latencies,95)\n",
    "    return f\"P95 latency (ms) - {time_p95_ms}; Average latency (ms) - {time_avg_ms:.2f} +\\- {time_std_ms:.2f};\", time_p95_ms\n",
    "\n",
    "\n",
    "vanilla_model=measure_latency(vanilla_clf)\n",
    "quantized_model=measure_latency(q8_clf)\n",
    "\n",
    "print(f\"Vanilla model: {vanilla_model[0]}\")\n",
    "print(f\"Quantized model: {quantized_model[0]}\")\n",
    "print(f\"Improvement through quantization: {round(vanilla_model[1]/quantized_model[1],2)}x\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We managed to accelerate our model latency from 68.4ms to 27.55ms or 2.48x while keeping 99.72% of the accuracy. \n",
    "\n",
    "![performance](assets/performance.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Push the quantized model to the Hub\n",
    "\n",
    "The Optimum model classes like `ORTModelForSequenceClassification` are integrated with the Hugging Face Model Hub, which means you can not only load model from the Hub, but also push your models to the Hub with `push_to_hub()` method. That way we can now save our qunatized model on the Hub to be for example used inside our inference API.\n",
    "\n",
    "_We have to make sure that we are also saving the `tokenizer` as well as the `config.json` to have a good inference experience._\n",
    "\n",
    "If you haven't logged into the `huggingface hub` yet you can use the `notebook_login` to do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from huggingface_hub import notebook_login\n",
    "\n",
    "notebook_login()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we have configured our hugging face hub credentials we can push the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/miniconda3/lib/python3.9/site-packages/huggingface_hub/hf_api.py:79: FutureWarning: `name` and `organization` input arguments are deprecated and will be removed in v0.8. Pass `repo_id` instead.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoTokenizer\n",
    "from optimum.onnxruntime import ORTModelForSequenceClassification\n",
    "\n",
    "tmp_store_directory=\"onnx_hub_repo\"\n",
    "repository_id=\"distilbert-onnx-banking77\"\n",
    "\n",
    "model.save_pretrained(tmp_store_directory)\n",
    "tokenizer.save_pretrained(tmp_store_directory)\n",
    "\n",
    "model.push_to_hub(tmp_store_directory,\n",
    "                  repository_id=repository_id,\n",
    "                  use_auth_token=True\n",
    "                  )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Load and run inference with a quantized model from the hub\n",
    "\n",
    "This step serves as a demonstration of how you could use optimum in your api to load and use our qunatized model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1235fec9d2614cf1a59baf0fc95e90b2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/5.78k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e6fb8d28e78441d990e6a4226c9407ea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/173M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "54a0c297709f49d498b9383cd8ea65dc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/314 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1e7545a4491b4c5ea0a3c5a0986e0db9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/226k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9752e5524cad42b9a7640d4c7272e656",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/695k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "70b672c0b26f4de6861a1032591947d8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/112 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'label': 'exchange_rate', 'score': 0.9794749021530151}]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from optimum.onnxruntime import ORTModelForSequenceClassification\n",
    "from transformers import pipeline, AutoTokenizer\n",
    "\n",
    "model = ORTModelForSequenceClassification.from_pretrained(\"philschmid/distilbert-onnx-banking77\")\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"philschmid/distilbert-onnx-banking77\")\n",
    "\n",
    "remote_clx = pipeline(\"text-classification\",model=model, tokenizer=tokenizer)\n",
    "\n",
    "remote_clx(\"What is the exchange rate like on this app?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.37 ms ± 236 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "remote_clx(\"What is the exchange rate like on this app?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "We successfully quantized our vanilla Transformers model with Hugging Face and managed to accelerate our model latency  68.4ms to 27.55ms or 2.48x while keeping 99.72% of the accuracy. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "7a2c4b191d1ae843dde5cb5f4d1f62fa892f6b79b0f9392a84691e890e33c5a4"
  },
  "kernelspec": {
   "display_name": "Python 3.9.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}