notebooks/keras-with-openvino-backend/keras-with-openvino-backend.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5f675795-d1ad-4cbb-b8f1-f51c87f1b3dc",
   "metadata": {},
   "source": [
    "# Run inference in Keras 3 with the OpenVINO™ IR backend\n",
    "\n",
    "Starting with release 3.8, [Keras](https://github.com/keras-team/keras) provides native integration with the OpenVINO backend for accelerated inference. This integration enables you to leverage OpenVINO performance optimizations directly within the Keras workflow, enabling faster inference on OpenVINO supported hardware.\n",
    "\n",
    "\n",
    "In this tutorial, we will show how to run inference of an end-to-end [BERT model for classification tasks](https://www.kaggle.com/models/keras/bert/) using the OpenVINO backend.\n",
    "\n",
    "\n",
    ">**Note**: The OpenVINO backend may currently lack support for some operations. This will be addressed in upcoming Keras releases as operation coverage is being expanded.\n",
    "\n",
    ">**Note**: The `tensorflow-text` package [isn't provided for Windows after version 2.10](https://github.com/tensorflow/text#a-note-about-different-operating-system-packages). `tensorflow-text==2.16.1` - the last version that supports `macOS x86_64`, but it doesn't support `macOS arm` and `python3.12`. Since tensorflow-text==2.17.0 supports `macOS arm`, since `2.18.1` - `python12`. This package is required for `BertTokenizer`.\n",
    "\n",
    "\n",
    "#### Table of contents:\n",
    "\n",
    "- [Prerequisites](#Prerequisites)\n",
    "- [Load the model with the OpenVINO backend and inference](#Load-the-model-with-the-OpenVINO-backend-and-inference)\n",
    "- [Sentiment Classification Example](#Sentiment-Classification-Example)\n",
    "\n",
    "\n",
    "### Installation Instructions\n",
    "\n",
    "This is a self-contained example that relies solely on its own code.\n",
    "\n",
    "We recommend  running the notebook in a virtual environment. You only need a Jupyter server to start.\n",
    "For details, please refer to [Installation Guide](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide).\n",
    "\n",
    "<img referrerpolicy=\"no-referrer-when-downgrade\" src=\"https://static.scarf.sh/a.png?x-pxid=5b5a4db0-7875-4bfb-bdbd-01698b5b1a77&file=notebooks/keras-with-openvino-backend/keras-with-openvino-backend.ipynb\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "814330bf-625d-4c19-8ee9-a0f8d21c3fab",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f4b9c29-820e-4fa0-abdc-d9618362b119",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q \"openvino>=2025.0.0\"\n",
    "%pip install -q \"keras>=3.8\" \"keras-hub\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5b7aa41",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import requests\n",
    "\n",
    "\n",
    "if not Path(\"notebook_utils.py\").exists():\n",
    "    r = requests.get(\n",
    "        url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py\",\n",
    "    )\n",
    "\n",
    "    open(\"notebook_utils.py\", \"w\").write(r.text)\n",
    "\n",
    "\n",
    "# Read more about telemetry collection at https://github.com/openvinotoolkit/openvino_notebooks?tab=readme-ov-file#-telemetry\n",
    "from notebook_utils import collect_telemetry\n",
    "\n",
    "collect_telemetry(\"keras-with-openvino-backend.ipynb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef034f42-b4cf-4cdf-b02d-44954342b3a6",
   "metadata": {},
   "source": [
    "## Load the model with the OpenVINO backend and inference\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Keras provides list of pretrained for general purposes models that can be used for fine-tuning on specific task.\n",
    "\n",
    "We will use the BERT model using the [`BertTextClassifier`](https://keras.io/keras_hub/api/base_classes/text_classifier/#textclassifier-class) class. OpenVINO API provides only inference capabilities, which means that before moving to the OpenVINO backend, you need to train the model on your own data using one of the backends that supports training. Once your model training process is finished, you can move to OpenVINO for inference speedup. Here are the general steps you need for that:\n",
    "\n",
    "    1. Specify the backend using an environment variable.\n",
    "    2. Create a model instance.\n",
    "    3. Run model prediction.\n",
    "\n",
    "To switch to the OpenVINO backend in Keras 3, set the `KERAS_BACKEND` environment variable to `openvino` or specify the backend in the local configuration file at `~/.keras/keras.json`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a2c9140-361f-4135-a463-7b9a494d6747",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"KERAS_BACKEND\"] = \"openvino\"\n",
    "import numpy as np\n",
    "import keras_hub"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e76fa6ca-e3b2-42e1-98a9-af95e1ef2155",
   "metadata": {},
   "source": [
    "Create a model instance. Take a model from [KerasHub](https://keras.io/keras_hub/presets/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c1958a3-89c9-4fb9-b505-7f4129c274b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "bert = keras_hub.models.BertTextClassifier.from_preset(\n",
    "    \"bert_base_en_uncased\",\n",
    "    num_classes=4,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90642b50-c073-4010-bef4-89e4b4945b08",
   "metadata": {},
   "source": [
    "Run model prediction for raw string data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2139429d-c714-47b9-804a-aa0de39dee35",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 398ms/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[ 0.14641605,  0.33292952, -0.07132149,  0.2362039 ],\n",
       "       [ 0.14057046,  0.2972596 , -0.02436665,  0.29821312]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features = [\"The quick brown fox jumped.\", \"I forgot my homework.\"]\n",
    "\n",
    "bert.predict(x=features, batch_size=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ad2bc93-c548-45a6-9ac6-cca303d5b63c",
   "metadata": {},
   "source": [
    "Preprocessed integer data. You can obtain this data using any tokenizer. In the previous example, the default tokenizer was used to achieve this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4c21effe-b418-41e7-a236-373aa9d349d1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 446ms/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[-0.03224922,  0.09847151,  0.32198498,  0.09585449],\n",
       "       [-0.03224922,  0.09847151,  0.32198498,  0.09585449]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features = {\n",
    "    \"token_ids\": np.ones(shape=(2, 12), dtype=\"int32\"),\n",
    "    \"segment_ids\": np.array([[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0]] * 2),\n",
    "    \"padding_mask\": np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]] * 2),\n",
    "}\n",
    "\n",
    "bert = keras_hub.models.BertTextClassifier.from_preset(\n",
    "    \"bert_base_en_uncased\",\n",
    "    num_classes=4,\n",
    "    preprocessor=None,\n",
    ")\n",
    "\n",
    "predictions = bert.predict(x=features, batch_size=2)\n",
    "\n",
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "994ce7b6-7795-44ca-9be2-8524176f02be",
   "metadata": {},
   "source": [
    "## Sentiment Classification Example\n",
    "\n",
    "This example demonstrates how to use a pre-trained BERT model [bert_tiny_en_uncased_sst](https://www.kaggle.com/models/keras/bert/keras/bert_tiny_en_uncased_sst2) from [KerasHub](https://keras.io/keras_hub/presets/) to perform sentiment classification on a set of sentences. The model predicts whether each sentence expresses a positive or negative sentiment.\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2396fee9-22c8-479f-954b-dca91f0a4914",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 234ms/step\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 170ms/step\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 165ms/step\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 165ms/step\n",
      "Sentence                                           Sentiment\n",
      "-------------------------------------------------------------\n",
      "the movie was a complete waste of time.            negative\n",
      "the plot was predictable and boring.               negative\n",
      "i absolutely loved this movie, it was fantastic!   positive\n",
      "an excellent movie that i would highly recommend.  positive\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "\n",
    "bert = keras_hub.models.BertTextClassifier.from_preset(\n",
    "    \"bert_tiny_en_uncased_sst2\",\n",
    "    num_classes=2,\n",
    ")\n",
    "\n",
    "sentences = [\n",
    "    \"the movie was a complete waste of time.\",\n",
    "    \"the plot was predictable and boring.\",\n",
    "    \"i absolutely loved this movie, it was fantastic!\",\n",
    "    \"an excellent movie that i would highly recommend.\",\n",
    "]\n",
    "\n",
    "\n",
    "def get_sentiment(text):\n",
    "    predictions = bert.predict([text])\n",
    "    probabilities = tf.nn.softmax(predictions, axis=1).numpy()\n",
    "    sentiment = np.argmax(probabilities, axis=1)[0]\n",
    "    sentiment_label = \"positive\" if sentiment == 1 else \"negative\"\n",
    "\n",
    "    return sentiment_label\n",
    "\n",
    "\n",
    "def display_results(results):\n",
    "    max_sentence_length = max(len(result[\"Sentence\"]) for result in results)\n",
    "    max_sentiment_length = max(len(result[\"Sentiment\"]) for result in results)\n",
    "\n",
    "    print(f\"{'Sentence':<{max_sentence_length}}  {'Sentiment':<{max_sentiment_length}}\")\n",
    "    print(\"-\" * (max_sentence_length + max_sentiment_length + 4))\n",
    "\n",
    "    for result in results:\n",
    "        sentence = result[\"Sentence\"]\n",
    "        sentiment = result[\"Sentiment\"]\n",
    "        print(f\"{sentence:<{max_sentence_length}}  {sentiment:<{max_sentiment_length}}\")\n",
    "\n",
    "\n",
    "results = []\n",
    "for sentence in sentences:\n",
    "    sentiment = get_sentiment(sentence)\n",
    "    results.append({\"Sentence\": sentence, \"Sentiment\": sentiment})\n",
    "\n",
    "\n",
    "display_results(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a789ff74",
   "metadata": {
    "test_replace": {
     "# %pip uninstall -q -y \"tensorflow-cpu\" tensorflow keras": "%pip uninstall -q -y \"tensorflow-text\" \"tensorflow-cpu\" tensorflow keras\n%pip install tensorflow\n%pip uninstall -y tensorflow"
    }
   },
   "outputs": [],
   "source": [
    "# Cleanup\n",
    "# %pip uninstall -q -y \"tensorflow-cpu\" tensorflow keras"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "openvino_notebooks": {
   "imageUrl": "https://github.com/user-attachments/assets/f0f17c2e-0f00-49b5-8364-8d18f2cd06ff",
   "tags": {
    "categories": [
     "API Overview"
    ],
    "libraries": [],
    "other": [],
    "tasks": [
     "Text Classification"
    ]
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}