notebooks/explainer_examples_v2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d985c392",
   "metadata": {},
   "source": [
    "# Example model explanations with Seldon  and v2 Protocol - Incubating\n",
    "\n",
    "In this notebook we will show examples that illustrate how to explain models using [MLServer] (https://github.com/SeldonIO/MLServer).\n",
    "\n",
    "MLServer is a Python server for your machine learning models through a REST and gRPC interface, fully compliant with KFServing's v2 Dataplane spec. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9da1204",
   "metadata": {},
   "source": [
    "## Running this Notebook\n",
    "\n",
    " This should install the required package dependencies, if not please also install:\n",
    " \n",
    "- install and configure `mc`, follow the relevant section in this [link](https://docs.seldon.io/projects/seldon-core/en/latest/examples/minio_setup.html)\n",
    "\n",
    "- run this jupyter notebook in conda environment\n",
    "```bash\n",
    "$ conda create --name python3.8-example python=3.8 -y\n",
    "$ conda activate python3.8-example\n",
    "$ pip install jupyter\n",
    "$ jupyter notebook\n",
    "```\n",
    "\n",
    "- instal requirements\n",
    " - [alibi package](https://pypi.org/project/alibi/)\n",
    " - `sklearn`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf7c5645",
   "metadata": {
    "id": "3e3c2c07-47ea-40b6-a3b2-5822c92d13f0"
   },
   "outputs": [],
   "source": [
    "!pip install sklearn alibi"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "591c69b6",
   "metadata": {},
   "source": [
    "## Setup Seldon Core\n",
    "\n",
    "Follow the instructions to [Setup Cluster](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html#Setup-Cluster) with [Ambassador Ingress](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html#Ambassador) and [Install Seldon Core](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html#Install-Seldon-Core).\n",
    "\n",
    " Then port-forward to that ingress on localhost:8003 in a separate terminal either with:\n",
    "\n",
    " * Ambassador: `kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080`\n",
    " * Istio: `kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8003:8080`\n",
    "\n",
    "### Setup MinIO\n",
    "\n",
    "Use the provided [notebook](https://docs.seldon.io/projects/seldon-core/en/latest/examples/minio_setup.html) to install Minio in your cluster and configure `mc` CLI tool. \n",
    "Instructions [also online](https://docs.seldon.io/projects/seldon-core/en/latest/examples/minio_setup.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f4a7085",
   "metadata": {},
   "source": [
    "## Train `iris` model using `sklearn`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41bf4ce1",
   "metadata": {
    "id": "828a7384-acdb-45fe-bed7-31e74693955d"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import shutil\n",
    "\n",
    "from joblib import dump\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d658769e",
   "metadata": {},
   "source": [
    "### Train model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "029ba0eb",
   "metadata": {
    "id": "8abc410e-8f82-46d3-a001-4bb787656609"
   },
   "outputs": [],
   "source": [
    "iris_data = load_iris()\n",
    "\n",
    "clf = LogisticRegression(solver=\"liblinear\", multi_class=\"ovr\")\n",
    "clf.fit(iris_data.data, iris_data.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "719842c1",
   "metadata": {},
   "source": [
    "### Save model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8757d345",
   "metadata": {
    "id": "c587bcaa-a792-4124-a789-e04fc34e5438"
   },
   "outputs": [],
   "source": [
    "modelpath = \"/tmp/sklearn_iris\"\n",
    "if os.path.exists(modelpath):\n",
    "    shutil.rmtree(modelpath)\n",
    "os.makedirs(modelpath)\n",
    "modelfile = os.path.join(modelpath, \"model.joblib\")\n",
    "\n",
    "dump(clf, modelfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d67818f2",
   "metadata": {},
   "source": [
    "## Create `AnchorTabular` explainer "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c369a479",
   "metadata": {},
   "source": [
    "### Create explainer artifact"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2dcaf4c0",
   "metadata": {
    "id": "aa84285d-35b0-4569-8741-65cb7be7a921"
   },
   "outputs": [],
   "source": [
    "from alibi.explainers import AnchorTabular\n",
    "\n",
    "explainer = AnchorTabular(clf.predict, feature_names=iris_data.feature_names)\n",
    "explainer.fit(iris_data.data, disc_perc=(25, 50, 75))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcb9338e",
   "metadata": {},
   "source": [
    "### Save explainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c854f554",
   "metadata": {
    "id": "4180f907-76e3-418b-a51f-984dca5f26ad"
   },
   "outputs": [],
   "source": [
    "explainerpath = \"/tmp/iris_anchor_tabular_explainer_v2\"\n",
    "if os.path.exists(explainerpath):\n",
    "    shutil.rmtree(explainerpath)\n",
    "explainer.save(explainerpath)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20b490f9",
   "metadata": {},
   "source": [
    "## Install dependencies to pack the enviornment for deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2bd4524",
   "metadata": {
    "id": "290b1f71-9c3a-406e-a744-6b7b9b5e7bf1"
   },
   "outputs": [],
   "source": [
    "pip install conda-pack mlserver==0.6.0.dev2 mlserver-alibi-explain==0.6.0.dev2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fb3f64d",
   "metadata": {},
   "source": [
    "## Pack enviornment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "482071c2",
   "metadata": {
    "id": "e77cb5a6-ff66-4582-a305-6b63f47d33c4"
   },
   "outputs": [],
   "source": [
    "import conda_pack\n",
    "\n",
    "env_file_path = os.path.join(explainerpath, \"environment.tar.gz\")\n",
    "conda_pack.pack(\n",
    "    output=str(env_file_path),\n",
    "    force=True,\n",
    "    verbose=True,\n",
    "    ignore_editable_packages=False,\n",
    "    ignore_missing_files=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b33bfef3",
   "metadata": {},
   "source": [
    "## Copy artifacts to object store (`minio`)\n",
    "\n",
    "### Configure `mc` to access the minio service in the local kind cluster\n",
    "note: make sure that minio ip is reflected properly below, run:\n",
    "- `kubectl get service -n minio-system`\n",
    "- `mc config host add minio-seldon [ip] minioadmin minioadmin`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b4b277b",
   "metadata": {
    "id": "125e0c80-98ed-4bf0-87e3-990b98ca433b"
   },
   "outputs": [],
   "source": [
    "target_bucket = \"minio-seldon/models\"\n",
    "os.system(f\"mc rb --force {target_bucket}\")\n",
    "os.system(f\"mc mb {target_bucket}\")\n",
    "os.system(f\"mc cp --recursive {modelpath} {target_bucket}\")\n",
    "os.system(f\"mc cp --recursive {explainerpath} {target_bucket}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48d5eef5",
   "metadata": {},
   "source": [
    "## Deploy to local `kind` cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebd83919",
   "metadata": {},
   "source": [
    "### Create deployment CRD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d7548d2",
   "metadata": {
    "id": "527c3ddd-a11c-4978-a457-47ecba5498e3"
   },
   "outputs": [],
   "source": [
    "%%writefile iris-with-explainer-v2.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: iris\n",
    "spec:\n",
    "  protocol: kfserving  # Activate v2 protocol / mlserver usage\n",
    "  name: iris\n",
    "  annotations:\n",
    "    seldon.io/rest-timeout: \"100000\"\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: SKLEARN_SERVER\n",
    "      modelUri: s3://models/sklearn_iris\n",
    "      envSecretRefName: seldon-rclone-secret\n",
    "      name: classifier\n",
    "    explainer:\n",
    "      type: AnchorTabular\n",
    "      modelUri: s3://models/iris_anchor_tabular_explainer_v2\n",
    "      envSecretRefName: seldon-rclone-secret\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62f79d0f",
   "metadata": {},
   "source": [
    "### Deploy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "120475cc",
   "metadata": {
    "id": "6ffd9063-3a2c-471c-af77-0c8255406075"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/iris created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f iris-with-explainer-v2.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e3038ca",
   "metadata": {
    "id": "03274f1a-65d0-4ba7-8b59-ded2be18fa8b"
   },
   "outputs": [],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=iris -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "268d97b3",
   "metadata": {},
   "source": [
    "### Test explainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d27de524",
   "metadata": {
    "id": "f2b29e0a-6a98-4de5-9adc-52f625ebcb39"
   },
   "outputs": [],
   "source": [
    "!pip install numpy requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbba7464",
   "metadata": {
    "id": "a7859f55-1a34-4ea3-b400-6a436bb89694"
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "import numpy as np\n",
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6db9e7dc",
   "metadata": {
    "id": "f86f3b26-a5d1-454f-889c-68364e6fe0d1"
   },
   "outputs": [],
   "source": [
    "endpoint = \"http://localhost:8003/seldon/seldon/iris-explainer/default/v2/models/iris-default-explainer/infer\"\n",
    "\n",
    "test_data = np.array([[5.964, 4.006, 2.081, 1.031]])\n",
    "\n",
    "inference_request = {\n",
    "    \"parameters\": {\"content_type\": \"np\"},\n",
    "    \"inputs\": [\n",
    "        {\n",
    "            \"name\": \"explain\",\n",
    "            \"shape\": test_data.shape,\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": test_data.tolist(),\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "    ],\n",
    "}\n",
    "response = requests.post(endpoint, json=inference_request)\n",
    "\n",
    "explanation = json.loads(response.json()[\"outputs\"][0][\"data\"])\n",
    "print(\"Anchor: %s\" % (\" AND \".join(explanation[\"data\"][\"anchor\"])))\n",
    "print(\"Precision: %.2f\" % explanation[\"data\"][\"precision\"])\n",
    "print(\"Coverage: %.2f\" % explanation[\"data\"][\"coverage\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8edd6c8c",
   "metadata": {
    "id": "b80d6afd-3a70-47d7-8a2d-f81a63e00725"
   },
   "outputs": [],
   "source": [
    "!kubectl delete -f iris-with-explainer-v2.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "174996d5",
   "metadata": {
    "id": "e41c9db1-c887-43f5-b93d-9c9ac42793db"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}