components/outlier-detection/cifar10/cifar10_outlier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cifar10 Outlier Detection\n",
    "![demo](./demo.png)\n",
    "\n",
    "In this example we will deploy an image classification model along with an outlier detector trained on the same dataset. For in depth details on creating an outlier detection model for your own dataset see the [alibi-detect project](https://github.com/SeldonIO/alibi-detect) and associated [documentation](https://docs.seldon.io/projects/alibi-detect/en/latest/). You can find details for this [CIFAR10 example in their documentation](https://docs.seldon.io/projects/alibi-detect/en/latest/examples/od_vae_cifar10.html) as well.\n",
    "\n",
    "\n",
    "Prequisites:\n",
    "\n",
    "  * [Knative eventing installed](https://knative.dev/docs/install/)\n",
    "    * Ensure the istio-ingressgateway is exposed as a loadbalancer (no auth in this demo)\n",
    "  * [Seldon Core installed](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/install.html) \n",
    "    * Ensure you install for istio, e.g. for the helm chart `--set istio.enabled=true`\n",
    "    \n",
    "    Tested on GKE and Kind with Knative 0.18 and Istio 1.7.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -r requirements_notebook.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ensure istio gateway installed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f ../../../notebooks/resources/seldon-gateway.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat ../../../notebooks/resources/seldon-gateway.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup Resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create namespace cifar10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile broker.yaml\n",
    "apiVersion: eventing.knative.dev/v1\n",
    "kind: broker\n",
    "metadata:\n",
    " name: default\n",
    " namespace: cifar10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create -f broker.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile event-display.yaml\n",
    "apiVersion: apps/v1\n",
    "kind: Deployment\n",
    "metadata:\n",
    "  name: hello-display\n",
    "  namespace: cifar10\n",
    "spec:\n",
    "  replicas: 1\n",
    "  selector:\n",
    "    matchLabels: &labels\n",
    "      app: hello-display\n",
    "  template:\n",
    "    metadata:\n",
    "      labels: *labels\n",
    "    spec:\n",
    "      containers:\n",
    "        - name: event-display\n",
    "          image: gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display\n",
    "\n",
    "---\n",
    "\n",
    "kind: Service\n",
    "apiVersion: v1\n",
    "metadata:\n",
    "  name: hello-display\n",
    "  namespace: cifar10\n",
    "spec:\n",
    "  selector:\n",
    "    app: hello-display\n",
    "  ports:\n",
    "  - protocol: TCP\n",
    "    port: 80\n",
    "    targetPort: 8080"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f event-display.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the SeldonDeployment image classification model for Cifar10. We add in a `logger` for requests - the default destination is the namespace Knative Broker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile cifar10.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: tfserving-cifar10\n",
    "  namespace: cifar10\n",
    "spec:\n",
    "  protocol: tensorflow\n",
    "  transport: rest\n",
    "  predictors:\n",
    "  - componentSpecs:\n",
    "    - spec:\n",
    "        containers:\n",
    "        - args: \n",
    "          - --port=8500\n",
    "          - --rest_api_port=8501\n",
    "          - --model_name=resnet32\n",
    "          - --model_base_path=gs://seldon-models/tfserving/cifar10/resnet32\n",
    "          image: tensorflow/serving\n",
    "          name: resnet32\n",
    "          ports:\n",
    "          - containerPort: 8501\n",
    "            name: http\n",
    "            protocol: TCP\n",
    "    graph:\n",
    "      name: resnet32\n",
    "      type: MODEL\n",
    "      endpoint:\n",
    "        service_port: 8501\n",
    "      logger:\n",
    "        mode: all\n",
    "        url: http://broker-ingress.knative-eventing.svc.cluster.local/cifar10/default\n",
    "    name: model\n",
    "    replicas: 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f cifar10.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the pretrained VAE Cifar10 Outlier Detector. We forward replies to the message-dumper we started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we configure `seldonio/alibi-detect-server` to use rclone for downloading the artifact. \n",
    "If `RCLONE_ENABLED=true` environmental variable is set or any of the environmental variables contain `RCLONE_CONFIG` in their name then rclone\n",
    "will be used to download the artifacts. If `RCLONE_ENABLED=false` or no `RCLONE_CONFIG` variables are present then kfserving storage.py logic will be used to download the artifacts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile cifar10od.yaml\n",
    "\n",
    "apiVersion: v1\n",
    "kind: Secret\n",
    "metadata:\n",
    "  name: seldon-rclone-secret\n",
    "  namespace: cifar10    \n",
    "type: Opaque\n",
    "stringData:\n",
    "  RCLONE_CONFIG_GS_TYPE: google cloud storage\n",
    "  RCLONE_CONFIG_GS_ANONYMOUS: \"true\"\n",
    "\n",
    "---            \n",
    "\n",
    "apiVersion: serving.knative.dev/v1\n",
    "kind: Service\n",
    "metadata:\n",
    "  name: vae-outlier\n",
    "  namespace: cifar10\n",
    "spec:\n",
    "  template:\n",
    "    metadata:\n",
    "      annotations:\n",
    "        autoscaling.knative.dev/minScale: \"1\"\n",
    "    spec:\n",
    "      containers:\n",
    "      - image: seldonio/alibi-detect-server:1.8.0-dev\n",
    "        imagePullPolicy: IfNotPresent\n",
    "        args:\n",
    "        - --model_name\n",
    "        - cifar10od\n",
    "        - --http_port\n",
    "        - '8080'\n",
    "        - --protocol\n",
    "        - tensorflow.http\n",
    "        - --storage_uri\n",
    "        - gs://seldon-models/alibi-detect/od/OutlierVAE/cifar10\n",
    "        - --reply_url\n",
    "        - http://hello-display.cifar10\n",
    "        - --event_type\n",
    "        - io.seldon.serving.inference.outlier\n",
    "        - --event_source\n",
    "        - io.seldon.serving.cifar10od\n",
    "        - OutlierDetector\n",
    "        envFrom:\n",
    "        - secretRef:\n",
    "            name: seldon-rclone-secret"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f cifar10od.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a Knative trigger to forward logging events to our Outlier Detector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%writefile trigger.yaml\n",
    "apiVersion: eventing.knative.dev/v1\n",
    "kind: Trigger\n",
    "metadata:\n",
    "  name: vaeoutlier-trigger\n",
    "  namespace: cifar10\n",
    "spec:\n",
    "  broker: default\n",
    "  filter:\n",
    "    attributes:\n",
    "      type: io.seldon.serving.inference.request\n",
    "  subscriber:\n",
    "    ref:\n",
    "      apiVersion: serving.knative.dev/v1\n",
    "      kind: Service\n",
    "      name: vae-outlier\n",
    "      namespace: cifar10\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f trigger.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the IP address of the Istio Ingress Gateway. This assumes you have installed istio with a LoadBalancer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CLUSTER_IPS = !(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')\n",
    "CLUSTER_IP = CLUSTER_IPS[0]\n",
    "print(CLUSTER_IP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optionally add an authorization token here if you need one.Acquiring this token will be dependent on your auth setup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TOKEN = \"Bearer <my token>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are using Kind or Minikube you will need to port-forward to the istio ingressgateway and uncomment the following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CLUSTER_IP=\"localhost:8004\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SERVICE_HOSTNAMES = !(kubectl get ksvc -n cifar10 vae-outlier -o jsonpath='{.status.url}' | cut -d \"/\" -f 3)\n",
    "SERVICE_HOSTNAME_VAEOD = SERVICE_HOSTNAMES[0]\n",
    "print(SERVICE_HOSTNAME_VAEOD)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "import requests\n",
    "from alibi_detect.od.vae import OutlierVAE\n",
    "from alibi_detect.utils.perturbation import apply_mask\n",
    "from alibi_detect.utils.visualize import plot_feature_outlier_image\n",
    "\n",
    "train, test = tf.keras.datasets.cifar10.load_data()\n",
    "X_train, y_train = train\n",
    "X_test, y_test = test\n",
    "\n",
    "X_train = X_train.astype(\"float32\") / 255\n",
    "X_test = X_test.astype(\"float32\") / 255\n",
    "print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)\n",
    "classes = (\n",
    "    \"plane\",\n",
    "    \"car\",\n",
    "    \"bird\",\n",
    "    \"cat\",\n",
    "    \"deer\",\n",
    "    \"dog\",\n",
    "    \"frog\",\n",
    "    \"horse\",\n",
    "    \"ship\",\n",
    "    \"truck\",\n",
    ")\n",
    "\n",
    "\n",
    "def show(X):\n",
    "    plt.imshow(X.reshape(32, 32, 3))\n",
    "    plt.axis(\"off\")\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "def predict(X):\n",
    "    formData = {\"instances\": X.tolist()}\n",
    "    headers = {\"Authorization\": TOKEN}\n",
    "    res = requests.post(\n",
    "        \"http://\"\n",
    "        + CLUSTER_IP\n",
    "        + \"/seldon/cifar10/tfserving-cifar10/v1/models/resnet32/:predict\",\n",
    "        json=formData,\n",
    "        headers=headers,\n",
    "    )\n",
    "    if res.status_code == 200:\n",
    "        return classes[np.array(res.json()[\"predictions\"])[0].argmax()]\n",
    "    else:\n",
    "        print(\"Failed with \", res.status_code)\n",
    "        return []\n",
    "\n",
    "\n",
    "def outlier(X):\n",
    "    formData = {\"instances\": X.tolist()}\n",
    "    headers = {\n",
    "        \"Alibi-Detect-Return-Feature-Score\": \"true\",\n",
    "        \"Alibi-Detect-Return-Instance-Score\": \"true\",\n",
    "        \"ce-namespace\": \"default\",\n",
    "        \"ce-modelid\": \"cifar10\",\n",
    "        \"ce-type\": \"io.seldon.serving.inference.request\",\n",
    "        \"ce-id\": \"1234\",\n",
    "        \"ce-source\": \"localhost\",\n",
    "        \"ce-specversion\": \"1.0\",\n",
    "    }\n",
    "    headers[\"Host\"] = SERVICE_HOSTNAME_VAEOD\n",
    "    headers[\"Authorization\"] = TOKEN\n",
    "    res = requests.post(\"http://\" + CLUSTER_IP + \"/\", json=formData, headers=headers)\n",
    "    if res.status_code == 200:\n",
    "        od = res.json()\n",
    "        od[\"data\"][\"feature_score\"] = np.array(od[\"data\"][\"feature_score\"])\n",
    "        od[\"data\"][\"instance_score\"] = np.array(od[\"data\"][\"instance_score\"])\n",
    "        return od\n",
    "    else:\n",
    "        print(\"Failed with \", res.status_code)\n",
    "        return []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normal Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "idx = 1\n",
    "X = X_train[idx : idx + 1]\n",
    "show(X)\n",
    "predict(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets check the message dumper for an outlier detection prediction. This should be false."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl logs -n cifar10 $(kubectl get pod -n cifar10 -l app=hello-display -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outlier Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "X_mask, mask = apply_mask(\n",
    "    X.reshape(1, 32, 32, 3),\n",
    "    mask_size=(10, 10),\n",
    "    n_masks=1,\n",
    "    channels=[0, 1, 2],\n",
    "    mask_type=\"normal\",\n",
    "    noise_distr=(0, 1),\n",
    "    clip_rng=(0, 1),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show(X_mask)\n",
    "predict(X_mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets check the message dumper for a new message. This should show we have found an outlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl logs -n cifar10 $(kubectl get pod -n cifar10 -l app=hello-display -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now call our outlier detector directly and ask for the feature scores to gain more information about why it predicted this instance was an outlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "od_preds = outlier(X_mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now plot those feature scores returned by the outlier detector along with our original image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_feature_outlier_image(od_preds, X_mask, X_recon=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tear Down"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl delete ns cifar10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}